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I. INTRODUCTION 


A. BACKGROUND 


Since the 1970’s avionics have rapidly progressed from pnmarily analog to 
primarily digital. A paradigm shift has occurred in which aircraft architectures have 
migrated from many highly specialized “Black boxes” loosely interconnected, to a 
system of general-purpose computers sharing many specialized tasks and vast amounts of 


data. Figure 1 shows the evolution of the different architectures. 













Independent Avionics Federated Avionics 
(40’s - 50’s) (60’s - 70’s) 


Common Integrated 
Processors 





Common Digital 
Modules 


Common Analog 





u 
(Supercomputers) 


Integrated Avionics Advanced Integrated Avionics 
(80’s - 90’s) (Post 2000) 


Figure 1. Evolution of Avionics Architectures. From Ref. [1]. 


During this same period of time computational speeds have increased from | Hz, 
to rates today that exceed 1GHz and show no signs of plateau. In fact, Moore’s Law 


states that computer speeds will double every eighteen months into the foreseeable future 


[Ref. 2]. Today’s military aircraft also have requirements to carry sensors that can 
receive, display and relay video in real time. With all of these advances and shifts in 
architecture philosophy one vital technology has lagged, the bandwidth of the 


interconnection architecture. 


In the early 1970's the Federated Architecture was bor, and the MIL-STD-1553 
data bus was introduced to military aircraft. The 1553 bus had a nominal throughput of 
1.0 megabyte per second (MBs) [Ref. 3]. Thirty years later 1553 is still used in most 
military aircraft even though processing speeds have increased by six orders of 
magnitude. While the military has lagged he Gviten sector with regards to introducing 
computers with cutting edge processing speed, it can be shown that the civilian sector is 
just as reticent when it comes to system bandwidth. fi 1964 the throughput of the “state 
of the art’ IBM System/360 was 1.25 MBs, over thirty years later this had grown to only 
4.5MBs an increase of only four times. Amdahl’s law of /O bandwidth required states 
“for every CPU instruction per second one bit of /O bandwidth is required [Ref. 4].” 
Clearly, this shows that there is a need for better I/O interconnects. 


B. REQUIREMENTS 


During the process of selecting a new avionics interconnect, the requirements for 
this interconnect must be defined. These requirements will involve both technical 


requirements and fiscal requirements. 


1. Technical Requirements 
The following paragraphs will outline the major technical requirements of any 


avionics system. While this list should not be considered complete, it will address most 
of the concerns of any system designer/program manager. 


a. Real Time and Deterministic 
Military aircraft are constrained by some requirements that commercial 


applications are not. First, they must have low latency. In the shared memory systems of 
the integrated avionics architectures of the 21° Century, processors will require low 
latency so that they may operate at peak efficiency without waiting for data. The Joint 
Advanced Stnke Technology Program (JAST) Avionics Architecture Definition states 
that “high latency interconnects often result in very low efficiency parallel processors. 
Recent experiments in commercial message passing parallel processor systems have 
demonstrated that even high latency interconnects can have a devastating effect on 
processor efficiency [Ref. 1].” 

In the Military, computer systems are by their very nature real time 
systems. High priority message traffic must be delivered on time even in the presence of 
low priority traffic. The Oxford Dictionary of Computing [Ref. 5] defines a real time 
system as follows: “A real time system is any system in which the time at which output 1s 
produced is significant. This is usually because the input corresponds to some movement 
in the physical world, and the output has to relate to that same movement.” A second 
requirement is a high level of determinism. The system must behave in a predictable 
manner. Dr. Phil Dowe of The University of Tasmania in Australia [Ref. 6] defines 


determinism as “one who’s future and past states can be predicted from a complete 


knowledge of the current state and the laws of nature.” The system must be able to 
deliver error-free information in a well-defined time frame. 

A military aircraft must be designed to operate in a combat environment. 
It is to be expected that at some time the aircraft will sustain damage. Since the avionics 
are such an integral part of the aircraft, involved in such things as flight controls, 
weapons systems and navigation, the data cannot be interrupted and alternate data paths 
must be available in the event that one or more “wires” is damaged. Also, if one of the 
processors on the bus begins to fail there must be a mechanism to detect and correct or 
ignore errors. All of this must occur in real time and in a deterministic fashion with as 
little impact of mission capability as possible. 


b. Throughput 
As the aircraft has an increased need for more information from high 


bandwidth systems, the throughput available over the bus must also increase. This high 
throughput is also required to minimize the latency of data passing between processors. 
In 1994 the JAST mission systems Integrated Product Team (IPT) came up with a series 


of throughput requirements for the avionics bus. These requirements are summarized in 


Table 1. 


Table |. Data Rate and Throughput Projections. From Ref. [1]. 
Application (Year Data Rate Projection Throughput 
2010) (per channel) Projection 
(includes 


B) 
IRST 120 - 200 Mbits/sec 
FLIR 
ADAS 
SIT Awareness 
Navigation 
Threat Warning 











































4-10 GOPS 
] - 2 GOPS 
1 - 4 GOPS 


150 - 700 Mbits/sec 
150 - 700 Mbits/sec 
150 - 700 Mbits/sec 

















RGHPRF 280 Mbits/sec a | 
ASLC + RGHPRF 280 Mbits/sec 2-15 GOPS 
200-800 Mbits a 


EW-EO (Missile SEE  ADAS 
Warning) ABOVE 
EW-C3 (Special 0.5 - 1.0 GOPS 
Receiver) 


EW-EO (Laser 50 - 100 Mbits/sec 50 - 100 MIPS 
Warning) 
__ E O ee ee 
Eee ee 2-15 _GOPS 
Boueyc a aa 5 - 11 GOPS 
Total CNI — suite TBD 30= 50' GOS 
(WBDL+GPS+IFF) 
* Normally done by specialized preprocessors 


EW-RF (RWR/ESM) ] -2 Gbits/sec 0.5 - 2.0 GOPS 


Clearly, based on this data, the required throughput of the interconnect will need 
to be at least 2 Gbits/sec. However, given the rapid increases in processor technology, 
and the need for the aircraft to be viable for thirty or more years, a more realistic goal 
should be 4 to 10 Gbits/sec. 


Cc. Protocol 
Different computers “speak” different languages. They are addressed by 


various protocols. One device might use the Small Computer Scalable Interface (SCSI) 
while another might be communicating in Internet Protocol (IP) while yet another could 
be streaming video over Asynchronous Transfer Mode (ATM). The interconnect must be 
able to handle all of these (and many others) with a minimum amount of overhead. 


d. Flow Control 
Memory in a contained system is a finite commodity. For each message 


that is sent there must be adequate memory resources available in the receiver to process 


the message. If this is not the case it is very possible that messages could get lost 


unbeknownst to the sender. This would cause confusion between the nodes and probably 
crash the system. 

Flow control is ensuring that the receiver has the required resources 
available to receive the message or the sender is notified somehow that resources are not 
available and some buffering at the sender must take place. The next generation system 
must have some form of flow control. 


yas Fiscal Requirements 
The military procurement system is constrained by the budget process. All 


procurement decisions must be made with the goal of getting the best product that meets 
the design requirements for the least amount of money. Avionics systems are not exempt 
from this requirement. The JAST Mission Systems Integrated Product Team’s Avionics 
Architecture Definition [Ref. 1] is quoted below. 
Affordability is of primary importance to the JAST. Therefore the JAST 
avionics architecture must be predominantly driven by cost considerations. 
The avionics architecture should seek to reduce life cycle costs, especially 
development costs. Affordability constraints require the architecture to 
Support an open system concept, insertion and use of commercial and 
openly available military technology/standards, and the reuse of software. 
Avionics now account for up to 30% of the total aircraft cost [Ref. 1]. This figure 
will continue to increase in the foreseeable future, and now can be viewed as an area 


where “smart” procurement decisions will produce large savings. 


a. Scalability, Supportability and Maintainability 
Another change in aircraft and their procurement has been the length of 


time that aircraft are expected to remain in service. Currently, the U.S. Military is flying 
aircraft that are expected to have a useful service life of thirty plus years. This means that 


military aircraft configurations are continually being changed and upgraded. Through the 


use of transformer coupling, the MIL-STD-1553 bus is very tolerant to the addition or 
subtraction of nodes. Any replacement to MIL-STD-1553 would also have to support the 
easy addition, removal and maintenance of any nodes on the bus. 

Not only will the aircraft be upgraded over time, but also a military 
aircraft in combat is continually changing its configuration throughout a combat sortie. 
The interconnection may have to interface with weapons that are themselves shared 
processors. High rates of data may be passed between the aircraft’s internal avionics and 
the external stores. These stores will go away, and the system must be able to make a 
seamless transition from one configuration to another. 

In order to facilitate ease of maintenance, as well as simplicity when 
upgrading, a bus with a minimum of wires would be desired. With fewer connections the 
probability of failure due to poor connections is reduced. Also the overall weight of the 
aircraft is reduced, allowing more of the weight budget to be allocated to weapons and 
fuel. Fewer adjacent wires also reduce the problems of unintentional electromagnetic 
interference. Finally, it would be good to eliminate one of the major timing problems 
associated with parallel busses, clock skew. Bus clock skew is where the bits become 
displaced from one another in time [Ref. 4]. This displacement can force the receiving 
node to reassemble and possibly reorder the data bits prior to processing, resulting in 
delays and bottlenecks. All of these factors imply that a serial bus would be the best 
candidate for the next generation aircraft. 


b. Commercial Off the Shelf 
Currently there are two philosophies that the government has adapted in 


order to maximize savings. The first is through the use of Commercial Off the Shelf 


technology (COTS). The use of COTS greatly reduces the government’s development 
cost, as well as providing a path for future upgrades. Use of COTS also requires some 
use of caution. There may be many hidden costs associated with COTS. Prior to 
procurement, careful analysis must be completed to ensure that the COTS part meets the 
needs of the government. These requirements include but are not limited to 
environmental constraints, interoperability between systems and maintainability. 


C. Open Systems 
The second philosophy used is the Open Systems Approach. The Naval 


Sea Systems Command (NAVSEA) Acquisition Support [Ref. 7] office defines an open 
system as follows: 


An Open System is a system that implements sufficient open 
(vendor independence, non-proprietary, publicly available, and widely 
accepted) specifications and standards for interfaces, services, and 
supporting formats to enable properly engineered components to be 
utilized across a wide range of systems with minimal changes, to 
interoperate with other components on local and remote systems, and to 
interact with users in a style that facilitates portability. An Open System is 
characterized by: 


* Well defined, widely used, non-proprietary interfaces/protocols. 

e The use of standards which are developed/adopted by industry 
recognized standards bodies. 

° All aspects of system interfaces defined to facilitate new or 
additional systems capabilities for a wide range of applications. 

e Explicit provision for expansion or upgrading through the 


incorporation of additional or higher performance elements with minimal 
impact on the system. 


Conversely, NAVSEA defines closed systems in the following paragraph: 


Closed systems are regarded to be proprietary, secret, or patented, while 
open systems are based on standards which are agreed to and published by 
an accredited, consensus-based group. USD (A&T)'s memorandum of 29 
November 1994 states that "open system specifications and standards are 
consensus-based public or nonproprietary specifications and standards for 
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systems and interfaces of hardware, software, tools, and architecture. 
Acquisition and upgrade costs will be minimized through the use of open systems 


architecture. 


c. SUMMARY 
Most of the technical requirements (as defined by the JAST Program Office) of 


the next generation avionics interconnection are summarized in the following table. 


Table 2. Digital Network Requirements. From Ref. [1]. 


Data Rate-per path 












Apart from the technical requirements some decisions are made based on fiscal 
concerns. First, the interconnect should be a commercially viable solution. It should be 
widely implemented and a fairly mature technology. This will allow the use of COTS, 
which will in tum increase the number of vendors available and drive down the 
component price. Second, an open systems approach should be used. This means that a 


well-defined, non-proprietary standard should be at the heart of the system. 
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Il. FIBRE CHANNEL AS A SOLUTION 


A. FIBRE CHANNEL BACKGROUND 


Currently there are several candidates for the interconnect technology of the 21° 
Century aircraft. This section will address how well Fibre Channel meets each of the 
requirements laid out in section one. Section three will compare Fibre Channel with 


some of the other interconnect technologies. 


Computer processors are very fast. Application software is composed of many 
Mbytes of data. Unfortunately, no matter how fast the processor, the slowest process is a 
major factor in the overall time it takes the computer to complete a task. Typically that 
process 1s data access. The current computer interconnect technologies cannot supply 
data fast enough to the processor. As was noted earlier, the bandwidth of interconnect 
technology has only increased by factors of less than ten while the processors are 
operating at speeds that have increased by several orders of magnitude. All this leads to 
bottlenecks in the computer system. Fibre Channel was designed to address this shortfall 
[Ref. 4]. The major problem to be solved was a way to get large amounts of data from 
various remote “warehouses” of data storage elements (hard drives, tape or other media) 
to a processor running an application at a different site. Kembel [Ref. 4] states that, “... 
what was needed was a physical interface that was independent of any specific command 
set behavior and flexible enough to support different command sets and their associated 


ae! 


architectures.” In 1988 a study was begun on developing a standard that would address 


these concerns. 


I} 


B. FIBRE CHANNEL DESIGN 


The following paragraphs will discuss the relevant design points of Fibre Channel 
which may or may not make it suitable for use in an avionics environment. 


1. Serial Interface 
“Serial done nght is faster and cheaper than parallel outside the chip.” (Ed Lee, 


Fibre Channel Group, Ref. 8) Fibre Channel is a serial interface. Serial busses have 
many advantages over their parallel counter parts. Summarized in Table 3 are some of 
the tradeoffs between serial and parallel interfaces. 


Table 3. Parallel vs. Serial. After Ref. [4]. 
Advantage Parallel Advantage Serial 


Wider data bus Less complex cables and 
connectors. Less weight, 
less COSt. 


circuits. 
density improvements. 


Lower probability of 
failure due to decreased 


number of parts and 
connections. 


No cross talk on adjacent 
wires. 


No clock skew. (As clock 
speed inc. clock skew 
becomes more of a factor.) 
No cross talk (Faster clock 
pulses have more cross talk 
and noise.) 


transmit/receive distance. 
overhead (arbitration etc.) 
factor of loading. 











Each clock pulse clocks in 
more bits 


























Faster clock rate More clock pulses per unit 
time equals more bits per 


unit time. 




















Multi-drop shared bus Lower cost 


Up until recently, the advantage that made up for the increased complexity of the 
parallel] bus was that for the same data rate the serial bus had to operate N times faster, 
where N is the bus width. Today’s advances in semiconductor technology have allowed 
cost effective serial transmissions of over one gigabit per second [Ref. 4]. 


Ze High Throughput 
The current Fibre Channel standard, FC-PH full speed, is capable of operating at 


throughputs of one gigabit per second. Double-speed (200 Mbytes/sec) and quad-speed 
(400 Mbytes/sec) are being demonstrated today. [Ref. 4] | Future developments in 
technologies such as Dense Wave Division Multiplexing (DWDM) could increase speeds 
to greater than 40 gigabits per second. Throughputs such as these are more than adequate 
to address the needs of future avionics systems, and show that Fibre Channel meets the 
throughput criteria of the 21*' century aircraft. 


3: Scalability 
Fibre Channel is a full duplex bus that uses two cables between nodes of the 


system. One cable is the transmit fiber the other the receive fiber. Each set of two fibers 
and their associated connectors are refereed to as a link [Ref. 4]. Fibre Channel is 
designed to support three types of interconnections or links between subsystems, 


point-to-point, arbitrated loop and fabric. Examples of these are shown in Figure 2. 


Five Problems Fibre Channel Solves 


Fire Channel 
arOctrated lacp 





S)) Ging serves farms a high-epeed pee 4) — tsk 


Figure 2. Fibre Channel Topologies. From Ref. [9] 
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a. Point-to-Point Links 
The first and simplest interconnection is the point-to-point link shown in 


Figure 3. 


Figure 3. Point-to-Point Configuration. From Ref. [10]. 


The point-to-point only supports connection of two systems together. 
Scalability is impossible. A system cannot be added unless one of the two “old” systems 
is replaced with the “new” system. Also, there is no sharing of resources other than 
between the two connected systems. Clearly the point-to-point link is not a suitable 
interconnection scheme for the advanced integrated avionics architecture envisioned for 
the next generation aircraft. 


b. Arbitrated Loop 
Another way to connect Fibre Channel nodes together is by using an 


arbitrated loop (FC-AL). FC-AL was designed as a relatively inexpensive way for small 
companies to convert their remote storage to Fibre Channel. Two of the primary FC-AL 


topologies are the arbitrated loop ring and the arbitrated loop with hub. 
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Figure 4. Arbitrated Loop Ring Network. From Ref. [10]. 


In this configuration messages are passed from sub-system to subsystem until the 
message or data gets to the appropriate end user. 


Figure 5 shows the arbitrated loop with a central hub. 





Figure 5. Arbitrated Loop Ring Network With Central Hub. From Ref. [10]. 


The hub in the center can be configured to allow for communications to continue 
even if one subsystem on the loop is incapacitated for some reason. Both configurations 
are scalable, and can have up to 127 systems on one loop. Since bandwidth 1s shared a 
more reasonable figure is 20 to 30 devices. It is however not truly “hot-swappable.” Any 


time a device is added or removed from the topology the network must re-initialize 
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[Ref. 8]. Ed Lee of the Fibre Channel] Group is quoted, ““When compared to FC Switched 
Fabric, its sole benefit is its lower purchase price.” [Ref. 8] 


c. Switched Fabric 
Fibre Channel’s Switched Fabric (FC-SW) is a routing structure that 


examines the destination address 1n the header of the message and delivers the message to 
the intended recipient [Ref 4.]. There is no single path from one device to another, rather 
many possible paths of which one is chosen. Figure 6 shows one possible fabric 


configuration. 





Figure 6. Switched Fabric topology. From Ref. [10]. 


The fabric is the most flexible of all the topologies. First the number of 
possible nodes is greater that 16 million, giving unparalleled scalability. (Compare this 
to the 31 possible devices on a MIL-STD-1553B bus [Ref. 2].) Also, unlike FC-AL, true 
live-insertion, plug and play is possible [Ref. 8]. Clearly, in terms of scalability, for 
avionics applications, the switched fabric is the best configuration. 


4. Deterministic and Real Time 
“The best way to ensure determinism and real time performance is through excess 


bandwidth.” Ed Lee, The Fibre Channel Group. 


a. Point-to-Point and Arbitrated Loop 
The point-to-point architecture can be both deterministic and real time. 


Dedicated bandwidth ensures that the connection 1s only dedicated to one task, so that 
task 1s sure to be completed. Unfortunately, there is not much need for a two-system 
architecture, so it has no real practical use in an avionics environment. 

It has been shown that FC-AL is moderately scalable. FC-AL has a 
reasonable address space, and from a strictly scalability standpoint could be used in small 
architecture applications. The main drawbacks for using FC-AL are determinism and 
real time performance. By its design, every device on the loop shares the bandwidth. 
The more devices that are in the architecture, the less excess bandwidth that exists, and 
hence the less likely that the task will get completed within a certain timeframe. Also, 
since there is only one way for messages to travel around the ning, additional protocols 
must be added to arbitrate who gets priority on the loop. This adds overhead and delays. 
Some members of the Fibre Channel industry have referred to FC-AL as the “Arbitrated 
Noose” [Ref. 8] because of these limitations. Lastly, each time a node is added or 
removed a Loop Initialization Protoco] (LIP) is started. This LIP causes additional 
unpredictable delays further reducing determinism. 


b. Switched Fabric 
Switched fabric can be configured so that multiple, dedicated 


“conversations” can be conducted between devices. Due to the many paths available, 
very little arbitration is required. Also, as a truly “hot-swappable” configuration, no 
additional protocol is executed when the configuration of the system is changed. This 
allows much more deterministic behavior when compared to FC-AL. Finally, in fabric 
the bandwidth between any two devices does not have to be shared, giving switched 
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fabric a marked advantage over FC-AL in terms of aggregate bandwidth. This superior 
deterministic behavior makes switched fabric the logical topology choice if Fibre 
Channel is the network backbone. 


5. Flow Control 
It takes finite time for a receiver to process incoming traffic. If a second frame 


arrives while that processing is taking place it must be placed into some sort of buffer 
awaiting processing. Since there is only a finite amount of memory (hence a finite 
amount of buffers), the receiver must be able to regulate the rate at which frames are sent 
from the sender. If this was not done, frames could be sent to a node/switch without 
sufficient resources and that frame would be lost with no record of its demise. Flow 
control allows the receiver to control the rate at which the sender may send frames. All 
of the Fibre Channel topologies offer two types of flow control: buffer-to-buffer and 
end-to-end. 

Both types of Fibre Channel flow control use a term called “Credit” to ensure that 
frames are only sent to nodes that have buffers to receive them. Just as there are two type 
of flow control there are two types of credit: buffer-to-buffer and end-to-end credit. 
Credit is granted during the login process and is based on the resources available in the 
nodes. 


a. Buffer-To-Buffer Flow Control 
Ensuring buffer-to-buffer flow control is mandatory for all frames. 


Buffer-to-buffer flow control simply ensures that the next node in the path has resources 
available to process the frame. For example, upon login a node has credit granted to it 
from the switch to which it is attached. The credit then decrements by one each time that 


the node then sends a frame. The node is free to send frames as quickly as it can up to 
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the point where this credit is exhausted. Credit is returned to the node through use of the 
Fibre Channel primitive R_RDY, which is explained in Chapter III. Buffer-to-buffer 
credit in no way assures that the final destination has the resources to process the frame. 
End-to-end flow control is responsible for that task. 


b. End-To-End Flow Control 
End-to-end flow control is a way to guarantee delivery of packets in 


Class 1, Class 2 and Class 6 messages. Much like buffer-to-buffer flow control, during 
the login process the receiving nodes grant end-to-end credit to the nodes that send them 
traffic. This means that end-to-end credit is maintained by a node for all other nodes with 
which it may communicate. Unlike buffer-to-buffer, having end-to-end credit with a 
receiving node ensures that ultimate receiver has resources to process the frame. 

Much like buffer-to-buffer flow control a mechanism exists to replenish 
the end-to-end credit. It is the ACK frame. Unlike the R_RDY, the ACK is a full frame. 
It is considered a Frame Type 0 (FT-0) otherwise known as a Link Control Frame. Link 
Control Frames are described in more detail in Chapter III. 


6. Reliable Transmission/Error Detection and Correction 
All the error detection and correction features mentioned below are available in 


any of the Fibre Channel topologies. 


a. 8b/10b Coding 
Unlike parallel interfaces, serial interfaces cannot send a clock. The clock 


signal is required to define the time frame in which individual bits are valid. Fibre 
Channel uses a coding scheme known as 8b/10b coding to reconstruct the clock signal at 
the receive end [Ref. 4]. In 8b/10b ten bits are coded for each byte of data. 8b/10b 


coding has several benefits that make it a desirable way to send information between 
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systems. First 1t ensures a balanced transmission. A balanced transmission 1s defined as 
a transmission that for an arbitrary byte boundary an equal number of ones and zeros are 
broadcast, and whose frequency is approximately that of the transmission clock. This 
balanced transmission ensures the elimination of DC and low frequencies that in turn 
ensures the elimination of electrical and thermal transients [Ref. 8]. Another benefit is 
that the circuitry required to reconstruct the transmit clock is greatly simplified, and 
hence much cheaper. Lastly, this data protocol is independent of the media used (copper, 
optical etc.), and distance traveled [Ref. 8]. 


b. Cyclic Redundancy Check 
A Cyclic Redundancy Check (CRC) is performed on each message 


transmitted through the system. The CRC uses the following 32-bit polynomial [Ref. 4]: 
XP? + X79 4 Xe XK KT KT + XP 4X + XP 4 KO 4X74 X74 X41 
The bit error rate (BER) for this polynomial is less than Oe [Ref. 4]. 
This equates to less than one error for each 15 minutes of operation at one Gbps [Ref. 8]. 
This BER should be more than sufficient for any avionics application. 


dis Multi-Protocol 
Fibre Channel is a data passing protocol. It is concerned only with the correct 


passing of data not the format of that data. This means that any protocol] can be 
supported by a Fibre Channel interconnection scheme. 


8. Commercial Off the Shelf 
Fibre Channel was designed by the commercial industry. Research indicates that 


while Fibre Channel is being widely adapted for the Storage Area Network (SAN) 
industry, very few current military or rugged industrial applications could be found. 


Several vendors of Fibre Channel technology were contacted at the Fibre Channel 
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Technology Conference (FTCT 99), and none of those had any plans to enter into the 
military market. The lack of a clear COTS market such as exists for technologies like 
MIL-STD-1553, RS-232 or ARINC 429 busses make future availability of COTS Fibre 
Channel hardware with military applications somewhat less likely. 


y: American National Standards Institute Standard 
The American National Standards Institute (ANSI) supports all Fibre Channel 


standards and projects. The following paragraph was taken from the ANSI web page 
feet. 11] 
The American National Standards Institute (ANSI) has served in 
its Capacity as administrator and coordinator of the United States 
private sector voluntary standardization system for 80 years. 
Founded in 1918 by five engineering societies and _ three 
government agencies, the Institute remains a private, nonprofit 
membership organization supported by a diverse constituency of 
private and public sector organizations. 

The technical group that ANSI charters to oversee the development of these 
standards is the T1l1 group {Ref.12]. All current standards and projects are listed at the 
T11 web site [Ref. 12]. 

ANSI is a widely recognized standard and therefore Fibre Channel is suitable for 
the open systems approach for acquisition of systems for government use. The standards 
(and their current stage of development) that are most suitable for military application are 


described in the paragraphs below. 


a. FC-PH/FC-FS 
FC-PH is the base document for Fibre Channel. It contains the description 


of the data format as well as the physical characteristics of Fibre Channel. The latest 


approved version of FC-PH 1s FC-PH-3. 
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In 1998 a decision was made to consolidate the relevant sections (framing and 
signaling) from FC-PH, FC-PH-2, and FC-PH-3 and associated errata, annexes and 
amendments into a single document [Ref. 12]. For this reason future documents will 
probably refer to the key document as FC-FS as opposed to FC-PH. 


b. FC-AE 
FC-AE is a proposed standard that would allow Fibre Channel to transport 


information between avionics subsystems. It is currently in development, and a draft of 
the standard is listed as Ref. 13. Since this standard is only in draft form, there is some 
risk that it will not develop as forecast. 


c. FC-SL 
FC-SL stands for the slotted loop configuration of Fibre Channel. It is in 


development for use in very specific aerospace applications. FC-SL is being designed to 
provide real time data using a loop topology. It is similar in function to MIL-STD- 
1553B. There are three primary differences between FC-SL and FC-AL. First, a master — 
element is designated that is responsible for creating time slots for the other ports on the 
loop. Second, there is no arbitration. The master device controls all access to the loop. 
Finally, slotted loop port is more complex and costly than an arbitrated loop port. [Ref. 4] 


G. CONCLUSION 


The background research on Fibre Channel yielded some promising results. It 
was Clearly designed for scalability as well as high-bandwidth, low-latency, real-time 
communications. The development process is being conducted by a national standards 
organization in an open manner. These aspects lead to the conclusion that Fibre Channel] 


could be used in an avionics architecture. 
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Once a qualitative assessment was made as to the suitability of Fibre Channel, a 
search was made for a tool to help design and evaluate avionics architectures using Fibre 
Channel as the interconnection protocol. Since the architecture of the 21° Century is 
supposed to utilize shared processing resources, it was determined that a tool that was 
able to conduct complex simulations of networks would be a good way to evaluate Fibre 


Channel. One such software package is OPNET Modeler, by OPNET Inc. 


The next three sections describe OPNET Modeler in detail, the adaptation of 
OPNET to Fibre Channel, and lastly how one would use this tool in evaluation of an 


avionics system. 
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fil. MODEL CONSTRUCTION 


A. OPNET MODELER 
OPNET Modeler was first demonstrated at MIT in 1987. It was designed to be 


predictive network management software, which can boost R&D productivity, improve 
product quality, and reduce time-to-market. OPNET Modeler contains many models of 
existing hardware, allowing quick construction, test and analysis of many types of 
networks and protocols including, but not limited to ATM, Ethernet, Fast Ethernet, 
HTTP, and TCP-IP. For this project the strength of OPNET Modeler was in the ability to 
guickly model any type of packet behavior at many levels of abstraction. This was 
possible by use of the three primary editors included in the OPNET Modeler package. 
These are the Network Editor, the Node Editor and the Process Editor. Three other 
editors were used to a lesser extent: the Packet Editor, the Link Editor, and the PDF 
Editor. 


1. Network Editor 
The Network Editor is a graphical representation of a communications network. 


Networks are composed of Node and Link objects (developed in the Node and Link 
Editors respectively), that are quickly configurable using drop and drag processes. In this 
Fibre Channel model, nodes represent processors, sensors, and displays, and the link is 
designed to emulate a full speed (1Gbps) Fibre Channel link. The Network Editor 
allowed rapid construction and test of various possible configurations that might be used 


in a next generation aircraft architecture. 


ray Node Editor 
The Node Editor captures the architecture of a network device or system by 


depicting the flow of data between functional elements, called "modules." Each module 
can generate, send, and receive packets from other modules to perform its function within 
the node. Modules typically represent applications, protocol layers, and physical 
resources, such as buffers, ports, and buses. Modules are assigned process models 
(developed in the Process Editor) to achieve any required behavior. In this simulation 
nodes were representative of three types of hardware: Dedicated Mission Processors, 
Displays and Sensors (which included antennas, radar, Electro-optic and navigation 
Sensors). 


Sh Process Editor 
Each module in a node has an underlying process that defines its behavior. These 


processes are defined using the Process Editor available in the OPNET Modeler software 
package. The process is presented to the user as a Finite State Machine (FSM). The 
FSM is composed of states and pathways. A packet enters the FSM and is acted upon or 
acts upon the system based on its contents and the state of the system. Tasks can thereby 
be simple for a low level of abstraction or highly complex making the model a more 
realistic representation of the actual system. 

Inside each of the states is code. This code is C/C++ making modifications easy 
and understandable to most system developers. Many common tasks, such as finding the 
value of a field in a packet, setting a value of a packet field etc. are already defined as 
functions available for use in OPNET. Also available in the Process Editor are Header 
Blocks, Function Blocks, State Variable Blocks and Temporary Vanable Blocks. Header 
Blocks contain macro definitions, global vanable definitions, function prototypes and 
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transition conditions. Function Blocks contain any user defined functions. These 
functions are automatically linked to the rest of the code through use of the OPNET 
compiler. State variables are used to maintain the status of a Node in between process 
invocations. These variables are created and defined in the State Variable Block. Unlike 
state variables, temporary variables do not maintain their value in between process 
invocations. Temporary variables are defined and in some cases initialized in the 
Temporary Variable Block. 


4. Link Editor 
The Link Editor allowed the specification of the Fibre Channel physical cabling. 


This is also known in the Fibre Channel Standard as the Physical Interface or FC-0. By 
using the Link Editor the Systems Designer is allowed to specify Bandwidth, Bit Error 
Rate (BER), link cost, propagation delay, packet types supported as well as other 
attributes of the Link. 


a Packet Editor 
The Packet Editor allows the System Designer to “build” packets of the format 


desired: in this case Fibre Channel Packets. In the Packet Editor one is able to name the 
sub-parts of a packet, specify the size of that packet (to include variable size data 
payloads), and specify the type of value accepted by that packet (i.e. integer, float, 
double, string etc.). 


6. PDF Editor 
The PDF Editor is a tool that allows the building of user specified Probability 


Density Functions (PDF). While OPNET Modeler has many predefined PDFs available 
for use, there are many circumstances where the user would like to use a PDF based 


either on previous test data or to test a theorem. 


B. OPNET MODELER IMPLEMENTATION OF FIBRE CHANNEL 


The following section will describe how OPNET Modeler was used to model the 
behavior of Fibre Channel in an avionics environment. It will describe the method used, 
the applicability to the Fibre Channel Standard, the data available and the limitations of 


the model. 


The Fibre Channel Model is based largely on the Fibre Channel Standard 
(FC-PH) [Ref. 12] as interpreted by Robert Kembel in his 1998 book, Comprehensive 
Introduction to Fibre Channel. Due to time constraints, many of the intricacies of the 
Fibre Channel Standard were not coded. The goal of the model was to have a fairly high 
level of abstraction, and yet still be able to make meaningful comments on its 
characteristics such as bandwidth, latency, and upgradeablity. Instances where there are 
deviations from the standard will be pointed out and their impact assessed. Furthermore 
the model was designed so that increases in detail and fidelity should be possible by 
future researchers. 


1. The Node 
All systems are comprised of nodes. For this model a node will simply be defined 


as a point from which data originates, is collected, displayed, processed or a combination 
of all of these elements. Examples of nodes in the system include antennas, mission 
computers, and displays. Since in this implementation the switch does not originate any 


of its own traffic or collect/process any other traffic, it will be discussed separately. 


In a broad sense any mission avionics system contains only three basic types of 
nodes: antennas, processors and displays. While there may be more subsystems like 
navigation, weapons, EW, EO etc, each of the components of these subsystems can be 
thought of in these terms. It is for this reason that in this model only three types of nodes 
were created. They are called: FC_ant, FC_disp, and FC_proc. Each node, no matter 
what type, can represented by four OPNET modules: a packet generator object, a 
processor, a receiver and a transmitter. Each component of the node will be descnbed in 


the following sections. The configuration is shown as Figure 7. 


1G) 
mn 


Figure 7. OPNET Implementation of a Fibre Channel Node. 


a. Packet Generator Object 
All nodes connected to a avionics system will generate message traffic 


across the Fibre Channel Fabric. Some will produce continuous streams of data 
(antennas), while others (primarily processors and displays) will absorb this data and 
generate response and command data. In order to model this behavior The OPNET 
Modeler Packet Generator Object was used. As described by the OPNET on-line help 


files, “The packet generator module is a stochastic packet source whose output can be 
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shaped by user-specified probability distributions. These distributions can control the 
frequency of packet arrivals and the length of packets.” The pnmary attribute of the 
Generator that was varied was the interarrival argument. This attnbute is in seconds and 
controls the output bandwidth. 


b. The Processor Module 
The processor module is the heart of each of the nodes. It contains the 


FSM (also known as the process model) that defines the behavior of the node. By 
construction only one FSM was designed. The design was such that based on the state of 
the node the FSM would take the appropriate path for that state. Since all nodes behave 
(in a Fibre Channel sense) in approximately the same manner, this design feature has 
minimal if any impact on simulation fidelity. The FSM and all its states will be examined 
in detail in Chapter IV. 


c. The Transmitter Module 
The transmitter is used to forward packets from the node onto the link 


connecting the node to the switch. There are very few adjustments that the designer has 
to make when using the OPNET provided transmitter object. The key attributes that have 
to be changed are the data rate and packets supported. For this model the data rate was 
set at 1,062,500,000 BPS, which represents full speed Fibre Channel operation. The 
model was required to support both R_RDY and FC_Header packets. These packets will 
be described in Section B.4.a and B.4.b. 


d. The Receiver Module 
The receiver is used to take packets off of the link from the switch and 


forward them to the node processor. The attributes for the receiver are the same as for 


the transmitter except that one additional attribute, ecc threshold, must be set. Ecc 
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threshold is used to define the highest proportion of bit errors allowed in a packet in 
order for the packet to be accepted by a receiver and forwarded to an output stream. For 
this model there was no attempt to model the effects of errors so ecce threshold was 
always Set to zero. 


2: The Switch 
The switch is the device used to connect nodes to each other. This includes 


connecting switches to other switches. The switch concept is fairly simple. Packets are 
received, the destination address is noted and the packet is forwarded to the appropriate 
destination. The switch modeled in this thesis is a 32 port switch. 32 represents the 
maximum number of links that can be connected to the switch. In a one-switch network 
32 nodes could be connected to each other. However when a fabric with multiple 
switches 1S created, more than one link can and should be used to connect switches to 
each other, so less than 32 nodes will be connected to each switch. Much like the node 
model, the switch is comprised of several basic components, one processor module, 
32 transmitter modules, and 32 receiver modules. The switch model is shown in 


Figure 8. 
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Figure 8. OPNET Model of the Fibre Channel Switch. 


a. The Processor Module 
Just like the node model, the processor module contains the logic required 


to receive and forward frames. Contained in the Processor Module is a FSM that defines 
the behavior of the switch. This FSM and its associated code will be defined in detail in 
Chapter IV. 


b. The Transmitter/Receiver Modules 
The 32 transmitter and receiver modules are identical to those used in the 


node model. 


i The Link Model 
All nodes must be connected by links. OPNET has a tool called the Link Editor 


that allows the designer to build appropriate behaviors into the desired link. For this 
Fibre Channel model the link was required to have a throughput of 1,062,500,000 BPS 
and support Fibre Channel frames as well] as Fibre Channel pnmitives. 
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4. The Packet Model 
Each packet sent across the link must be formatted. The Packet Editor was used 


to make the two types of packets used in this model, Fibre Channel Frames and Fibre 
Channel Primitives. 


a. The Fibre Channel Frame 
There are two types of frames in Fibre Channel, Data Frames and Link 


Contro] Frames. These are also referred to as FT-1 and FT-O respectively [Ref. 4]. The 
format for both FT-1 and FT-O frames is exactly the same for the first six words (a Fibre 
Channel word is defined as 32bits before 8b/10b encoding, and 40 bits after 8b/10b 
encoding). This section is referred to as the Frame Header. The difference between the 
two frame types comes after the header. Link Control Frames have no data payload 
whereas the Data Frames can have as much as 528 words of data after the header. Figure 


9 shows the OPNET Modeler representation of a Fibre Channel Frame. 
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Figure 9. OPNET Representation of a Fibre Channel Frame. 
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The OPNET packet model is broken up into the following pieces. 
First, every frame must be preceded and followed by at least three Fibre Channel 
primitives known as idles. So that bandwidth utilization and latency data would be 
accurate these 240 bits were added to the front (120 bits) and rear (120 bits) of the frame. 

The next piece of the frame is called the Start of Frame or SOF. The SOF 
identifies the beginning of the frame and contains information as to the Class of the frame 
that follows, whether to expect one frame or more than one frame as well as other high 
level information. In this model each frame was considered an independent piece of data 
and not part of some larger group (One analogy might be that this simulation models 
people shouting out random words that are not part of a sentence or paragraph.). For this 
reason the SOF was only used to identify what the class of frame was. 

After the SOF the next six words represent the Frame Header. In the 
model (Figure 9) the header is represented by the packets from R_Bits through 
Parameter. The header contains information about the originator of the message, the 
intended recipient, whether it is a FT-O or FT-1 as well as many other pieces of data that 
are required in a full Fibre Channel Protocol. Since many of the features of Fibre 
Channel were not modeled in this simulation, not all of these fields were used. However, 
in order to allow future users of this model to increase the fidelity of the model, most of 
the fields were dutifully represented. 

Following the header is the data payload. Though shown in Figure 9 as 32 
bits, the data payload is variable from 0 to 528 words. This variability allows for PDF 


modeling of data payloads based on experimentation/vendor data. 
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Several additional fields (sre id, sre subnet, and dst subnet) were added 
to aid in the simulation of some Fibre Channel behaviors. These fields have zero length 
and therefore do not affect any cnitical system parameters (specifically, bandwidth 
utilization and end-to-end delays). 

Next in the OPNET Fibre Channel Frame is the Cyclic Redundancy Check 
(CRC) field. The CRC is used to check the validity of the frame header and data field 
that were received. Since OPNET has the ability to flag frames with errors and then 
check for these flags, this field was not explicitly used. It does however affect the size of 
the frame, and therefore will accurately affect the bandwidth and latency characteristics 
of the system. 

The End of Frame (EOF) delimiter signals to the node that it may end the 
reception of the frame. The EOF also adds information as to the location of the frame in 
the larger context of the communication between the nodes. Using the conversation 
analogy, the EOF delimiter would represent spaces between words, punctuation or the 
end of a thought. Since this simulation does not encode at higher than the frame level, 
this capability is not used. 


b. The R_RDY Primitive 
The second type of packet used in this network is the R_RDY pnmitive. 


In Fibre Channel parlance a pnmitive is a word size (32 bits before 8b/10b encoding, 40 
bits after the encoding) packet used to indicate an event at the sending port [Ref. 4]. For 
this model] only one of the primitives is used, the R_RDY. The R_RDY is used for 


buffer-to-buffer flow control, and indicates that the intended receiver (node or switch) of 
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a frame has emptied a buffer and is ready to receive another frame. The OPNET 


representation of the R_RDY is shown in Figure 10. 


SS ae ——=; 
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Figure 10. OPNET Representation of a Fibre Channel Frame. 
The sre_nde field has zero length, and is only used to aid in routing of the 
R_RDY from its originator to the destination. It has no effect on simulation fidelity. 


A future improvement of this model would include the modeling of other 


primitives and noting their effect on the simulation. 
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IV. FINITE STATE MACHINE DESCRIPTIONS 


Ne OVERVIEW 
As was discussed earlier, the heart of the OPNET rendition of both the Fibre 


Channel Node and the Fibre Channel Switch is the Finite State Machine (FSM) contained 
in the processor module. It contains all the code that defines the behavior of the 
node/switch. Appendix B contains the code for the OPNET Fibre Channel model, and 
this chapter will describe each state in detail. 


B. THE NODE FINITE STATE MACHINE 


The node FSM is shown in Figure 11. The following paragraphs will trace each 
of the states and describe their function. Wherever deviations from the Fibre Channel 
Standard are programmed, the rationale behind this decision will be explained as well as 


what impact this design decision may have on simulation fidelity. 
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Figure 11. Fibre Channel Node Finite State Machine. 
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The node FSM is broken up into roughly four functional areas or paths, the 
initialization path, the packet generation path, the receive/process path, and the transmit 
path. Some states do no fit neatly into only one path. These states were put into the 
functional area that seemed to be the best fit. 


lie Initialization Path 
During initialization the node finds out who it is and what other nodes are in the 


network. Initialization takes place prior to any communications in the model. Several 
OPNET “housekeeping” chores are taken care of. These include the declaring of global 
variables, scheduling of interrupts, declaration and initialization of statistics, and 
initialization of the state of the node. The initialization path is composed of the following 
seven States: init, objid?, subnet?, function?, i talk to, computers, and others. 


a. Init State 
The init state is primarily used to initialize variables, register statistics set 


up the administration of the process. It does not affect the fidelity of the model. 


b. Objid? State 
In Fibre Channel there are levels of login: Fabric Login (FLOGI), Port 


Login (PLOGI) and Process Login (PRLI). In this model there are no upper level 
processes (FC-4). All communication occurs at the FC-2 level and below. For this 
reason PRLI is not addressed in the model. FLOGI is used by one node to establish 
communications with the fabric. PLOGI is used for one node to establish 
communications with another, specific node. Fibre Channel allows FLOGI and PLOGI to 
be done via explicit or implicit means. [Ref. 4] In this model each node learns the state 


of the fabric and the other nodes in the fabric via implicit means. That is to say, there are 
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no explicit Fibre Channel communications between nodes prior to passing of meaningful 
traffic. In order to facilitate this, each node must learn its object id or objid. 

The objid is a unique integer value given to all objects in the model at run 
time. It is a software attribute applicable to any OPNET Modeler model and has no 
parallel in Fibre Channel. Since each node address in the fabric must be unique, the 
objid was used to satisfy this requirement. The objid? state was used by the node to 
identify its own objid as well as to identify some of the attnbutes of the node. These 
attributes are Subsystem, Process Delay, and R_RDY Delay. The Subsystem attnbute 
identifies of what subsystem the node is a part. Subsystems include Radar, EW, EO, 
NAV and Navigation. Other subsystems could easily be added to more closely model 
existing or planned avionics architectures. Process Delay is used to model the time it 
takes for a processor to read, analyze and reply to a frame. R_RDY Delay is used to 
model the time required for a node to return an R_RDY. 


C. Subnet? State 
In order to route in OPNET both an objid and subnet are required. The 


subnet? state is used to determine the subnet of the node. This is an administrative state 
only and is only required so that the OPNET Modeler routing functions may be used. It 
neither adds nor subtracts from the fidelity of the simulation. 


d. Function? State 
There are several ways one could choose to design a distributed 


architecture. One way would be to have all processors available to run any task. Sensors 
would send their data for processing based solely on processor load. This has several 
drawbacks. First, processors would continuously have to load and unload the required 
application software in order to process the data required, thereby slowing down 
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processing time. Second, some account must be made for the cost of transporting the 
data through the fabnc. It could take much longer to send data to a light loaded processor 
4 or 5 hops away than to send it to a processor located on the same switch. 

Another way to design the architecture would be to have the processors 
and sensors grouped according to function. This would allow the software to remain 
“close by” in high speed RAM. The advantage of total distribution would still be there 
since any node in a fabric could talk to any other node provided that a redundant pathway 
exists. This would allow, for example, a primarily NAV based computer to process radar 
data in the event of battle damage. The fabric would simply have to route the required 
radar software and radar data to a NAV processor. 

This simulation was designed using the second philosophy. The 
function? state discovers the function and associated subsystem of the node. Since all 
the nodes have a standard “‘type name” (i.e. FC_ant, FC_proc, FC_disp), a string compare 
command was used to compare the “type” attribute with the standard names. A similar 
process was used to discover the node’s subsystem. 


é. I Talk To State 
The i talk to state is used to help the node discover what other nodes it 


will have to communicate with. A description of the logic follows. If the node was of 
type “FC_ant” the node is some kind of sensor and it could only be expected to talk to a 
processor. The assumption was make that sensors did not talk to other sensors except 
through an intermediate processor. Likewise, if the node was of type “FC_disp” it could 
only be expected to communicate with display processors. Lastly, if the node was of type 


“FC_proc”’ it could reasonably expect to talk to other processors, displays and sensors. 
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if Computers State 
The computers state is one of two states (others is the other one) which 


takes all the information gathered so far and builds a look up table of all the other nodes 
with which this node will communicate. During this state the node also gets other 
essential information about these nodes. Specifically, the amount of End-to-End credit 
available for use. These values are stored in state variable arrays. These variables are 
then available for use and modification during each process invocation. This state 
essentially fills the need for an implicit login. 


g. Others State 
The others state is nearly identical to the computers state, but is used 


when the node of interest is a processor. The reason for this is that an EW computer 
could very well have reason to communicate with the display group of processors. This 
state allows for processors from one subsystem to communicate with processors from 
other subsystems. It does not however allow a processor from one subsystem to talk to a 
sensor from another subsystem. 

The design does have some impact on the fidelity of the simulation. 
Currently there is no way to update the communication list in the event of battle damage. 
The consequence of this is that if, for example all of the radar processors were destroyed 
there would be no way (in the simulation) to divert the data to another processor. 


2. Traffic Generation Path 
The node uses the traffic generation path to generate its own traffic. This path is 


invoked when an interrupt from the ideal packet generator is received. Eight states are 
grouped into the traffic generation path. They are idle, get_pkt, b_to_b, b_plate, 


CL1_xmt, CL2_xmt, CL3_xmt, and e_to_e. 
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a. Idle State 
The idle state could easily be thought of in either the transmit or receive 


path. It is simply a state from which the process waits to be invoked. The idle state 
represents a node waiting to receive or send traffic. 


b. Get_pkt State 
In the node model packets come from one of two places, either the packet 


generator or the receiver. When the packet comes from the packet generator it must be 
assembled, a destination assigned, routed and sent. In this model the first step is the 
assigning of a destination. to accomplish this, the get_pkt state picks a uniformly 
distributed integer, which 1s then used as an index in the array of suitable destinations that 
was built in the computers or others state. A uniform PDF was chosen because the 
assumption was made for this model that communication with other nodes in subsystem 
group was equally likely. This distribution could be changed to fit any vendor/real- 
system data. 


Cc. B_to_b State 
In Fibre Channel there are two types of flow control, Buffer-to-Buffer and 


End-to-End. Both types are based on an incrementing/decrementing credit system. Each 
node has credit with other nodes in the system. These values were obtained during the 
implicit login that occurred in the computers or others state. Buffer-to-Buffer credit is 
designed to ensure that the next node in the routing through the fabric (in a fabric this 
will always be a switch) has enough space in memory/buffers to accept the frame. 

The b_to_b state is designed to check if there is buffer-to-buffer credit 
available. In reality if there is no credit available the frame must not be sent. It must 


somehow either be stored until credit is available (credit replenishment will be discussed 
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later) or destroyed. Currently the login Buffer-to-Buffer credit value is known implicitly 
by the node. In future designs it may be useful to discover this value based on the switch 
that is attached to the node of interest. This would allow the system designer/evaluator to 
place switches with known memory configurations in the system and evaluate the impact 
of delays and throughput. 

Since the purpose of this model is to evaluate designs, all packets are sent 
regardless of credit available, but the b_to_b state logs each time a packet is sent without 
credit and the credit is then replenished to its login value. This node statistic, known as 
‘“no_bb_count”, is then available to the system designer/evaluator. This deviation from 
the standard was not considered to have a significant impact on simulation fidelity. 


d. B_plate State 
There is a certain amount of similarity between frames in Fibre Channel. 


Much of the header field is common between frames originated in the same fabric. In the 
Navy things that are common and can be done in advance of any real work are called 
‘boilerplate’, hence the name of the state. 

The state b_plate simply sets some of the standard fields in the header of 
any frame that is sent. The values are taken from the standard as described on pages 179- 
206 of Robert Kembel’s book [Ref. 4]. Also, in this state the destination and source 
addresses are inserted in the frame header. These values were obtained in the objid? 
State. 

As a side note, there was a provision made to change the size of the data 
payload associated with each data frame. This function was rarely used during the test 


and analysis of various systems. The reasons are as follows. First, no effort was made to 
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model the Sequences and Exchanges associated with Fibre Channel. By using this 
approach, each node could be viewed as a frame-generating machine. Since the data field 
in Fibre Channel is relatively small (2.0625 Kbytes), it was assumed that most data being 
passed in any exchange would be greater than this (i.e. would require at least one, but 
probably more full-length frames). The result of this would be that the average data 
payload would approach the limit of 2.0625 Kbytes. After an analysis of a system it 
would be simple to vary the size of the data payload based on any PDF. This model and 
all data associated with it represent a worst case scenario. 


é. CLI_xmt State 
The CL1_xmt state is used to fill in fields that uniquely identify the frame 


as a Class | message. 


fe CL2_xmt State 
The CL2_xmt state is used to fill in fields that uniquely identify the frame 


as a Class 1 message. 


g. CL3_xmt State 
The CL3_xmt state is used to fill in fields that uniquely identify the frame 


as a Class 1 message. 


h. E_to_e State 
End-to-End credit is only applicable to Class 1, 2 and 6 frames. Since 


there is no delivery assurance in Class 3, only the Buffer-to-Buffer flow control criteria is 
required to be met prior to data transmission. The e_to_e state was designed to check for 
the availability of End-to-End credit prior to data transmission. 

The e_to_e state operates in a manner very similar to the b_to_b state in 
that all frames are sent regardless of the availability of credit. Once again, when the 
credit allocated at login is depleted a node statistic called “no_ee_count” is incremented 
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and the End-to-End credit is replenished. As with the b_to_b state, the value added by 
this statistic was deemed to more than offset the loss in simulation fidelity. 


3. Receive and Process Path 
The receive and process path is invoked upon receipt of a packet from the node’s 


receiver. This packet could be an R_RDY pnmitive, an FT-O or FT-1 frame. The receive 
and process path is comprised of six states. They are the rev, r_rdy_rcv, snd_rrdy, 
FC_0, FC_1 and check credit states. 


a. Rev State 
The rev state is responsible for determining whether the incoming packet 


is a Fibre Channel frame or a primitive. Since the only primitive defined for this model is 
the R_RDY, this state determines if the received packet is a frame or an R_LRDY. This 
task 1s accomplished by checking the number of fields in the incoming packet. Primitives 
only have one 40-bit field whereas frames have many more. 


b. R_rdy_rcv State 
The FSM associated with the node processor has to deal with two types of 


frames: frames that are produced by the node and frames that are received and processed 
by the node. In order to replenish the supply of Buffer-to-Buffer credit the system relies 
on receipt of a Fibre Channel primitive known as an R_RDY. Each time a switch 
receives any valid frame it returns an R_RDY to the node that forwarded the frame to it. 
Then, when the node receives the R_RDY, the Buffer-to-Buffer credit is incremented. 
The r_rdy_rec state is responsible for incrementing the buffer if it is less 
than full. If the buffer is already at its login value the buffer is left at the login value. 


The standard does not allow Buffer-to-Buffer credit to increase beyond its login value. 
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C. Snd_rrdy State 
If the node receives a frame from the switch it replies with an R_RDY 


primitive. The snd_rrdy state 1s responsible for constructing, routing and calculating the 
delay associated with sending the RLRDY. R_RDYs are sent immediately upon receipt 
of any frame, valid or not. The receipt of an R_RDY only indicates to the receiver that 
the resource or buffer is once again available to receive another frame. For this reason 
the R_RDY delay would be considerably faster than processing delays. Once again, an 
accurate simulation of a system would require the PDF of this delay to be defined. 
Currently an either an exponential PDF is used with the node attribute “R_RDY Delay” 
as the mean of the PDF, or the code is slightly modified to allow constant delays to be 
modeled. 


d. FC_O State 
The FC_0 state is entered if the received node is a LCF (FT-0) frame. 


Several tasks are accomplished. First the End-to-End credit associated with the sending 
node is incremented. Next, the time required to send the frame and receive the ACK 1s 
calculated and logged. This is a round tnp time. Round trp time is used instead of one 
way in Class 1, 2 and 6 communication because that is the time required to ensure that 
the data reached its intended recipient. If only one time was used as in the case of Class 
3, it would only represent the time required to send and receive a frame of unknown 
validity. Using this method ensures that if an ACK is received to a frame, the intended 
recipient did not only receive it, but also that it was valid and that task 1s complete. 
Another use of the FC_0 state is to determine if the LCF was an F_BSY 
instead of the expected ACK. When that occurs at least one of the switches on the path 


to the destination node or the destination node itself was either busy with Class 1 or 6 
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communications or did not have adequate resources to accept the frame. Each time an 
F_BSY is received a statistic “F_BSY_DAT” or “F_BSY_LCF’” is recorded. This 
Statistic helps analyze if adequate memory resources are available along various 
pathways. 

There are two types of FLBSY LCFs. The first type is in response to a 
FT-1 the other is in response to an FI-0. The purpose for differentiating between the two 
is to ensure that if a data frame was sent that the transmitting node 1s made aware that the 
frame did not reach its intended recipient. In order to allow flexibility to the system 
designer, Fibre Channel does not require a retransmit of the data. Secondly, so that 
endless “I’m busy to your busy of my busy etc. etc. etc” do no occur, F_BSYs are not 
returned when the frame that could not be sent was an F_BSY. 

The last function of the FC_0 state is to destroy the packet. This 
destruction is a key step in OPNET Modeler because it frees all the memory resources 
associated with that frame. Failure to destroy the packets slows down simulation and can 
cause the system to crash because of memory leaks. 


e. FC_1 State 
In Fibre Channel there are two types of frames defined. The first 1s Frame 


Type 0 or FT-0 which is a Link Control Frame (LCF). LCFs have a data length of zero 
and are used for administrative tasks. Frame Type 1 or FI-1 frames are data frames. 
During construction of the model a small typographical error was made and FC-1 was 
used in place of FT-1. This error is prevalent throughout the model. 

Upon receipt of a data frame the node uses the FC_1 state. If the 


incoming data frame is a Class 3 message, there is no need to assure the sender that the 
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frame was received, and the only tasks accomplished here are of a statistical nature. The 
time it took for the frame to transmit the fabric 1s recorded, and the frame is destroyed. 

If the incoming data frame is a Class 1, 2 or 6 frame a response from the 
node to the sender is required to assure the sender that the frame was received in tact. 
This is done through use of one of the FT-0 frame types called an ACK. In the FC_1 
state the ACK is constructed, and prepared for routing. 


af Check Credit State 
When it is required to return an ACK, the state check credit simply 


checks the value of the node’s Buffer-to-Buffer credit prior to sending the ACK back. If 
there is Buffer-to-Buffer credit available, it is decremented. If, however, it is zero it is 
treated as in the b_to_b state. In its value is replenished to the login value and the 
‘“no_bb_count” statistic 1s incremented. 


4. Transmit Path 
The transmit path 1s designed to calculate routes, update the network topology and 


send the packets out the receiver. Three states are included in this path. They are 
topo_build, route and xmt_pkt. 


a. Topo_build State 
In order to route frames through the fabric, OPNET Modeler must have a 


definition of the associated fabric or topology. The topo_build state is used to build and 
rebuild the topology of the fabric. This topology is stored as a global] pointer available to 
all nodes and switches in the fabric. This pointer contains information such as the cost of 
transiting each link, all object ids and subnets in the fabric. 

The function required to “build” the topology is very expensive 


computationally. In order to minimize simulation time and reduce memory usage. A 
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decision was made to rebuild the topology on a scheduled basis. This schedule is based 
solely on simulation time and not linked to any packet arrival at the processor. In this 
respect it should be considered an asynchronous event. This has some fidelity 
ramifications. First, the optimum path between two nodes is based on this topology 
pointer and the information associated with it. This fact means that between rebuilds of 
the topology the frame will take a less than optimum path. This is not unlike many 
systems in use today. Second, if nodes are disabled between topology rebuilds, frames 
may start out for a destination that 1s working when the frame leaves its source, but 
disabled by the time it arrives. The solution to increased levels of fidelity is to rebuild the 
topology more often with the limit being rebuilding the topology before each frame is 
transmitted. The tradeoff is increased memory usage and decreased simulation speed. 


b. Route State 
Any fabric has to have a algonthm to route frames from the source node to 


the destination node. Since the speed and efficiency of a system are normally of 
paramount concern to the user, and one measure of what makes one system “better” than 
another, most routing algorithms are proprietary in nature. This being the case, the 
routing algorithm used in this simulation was the default algorithm based in OPNET 
Modeler. The route state was designed to add a route pointer to the frame of interest. 
The routing algonthm was based solely on the cost of getting from source to destination. 
Cost was defined as a function of bandwidth usage. The higher the usage the higher the 
cost to transverse that link. By using this cost definition, bandwidth could be shared to 


the maximum extent possible. 
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Like topology building, route building is computationally expensive, and 
should be minimized. To that end, route was designed to first “check and see” if a route 
to the destination had already been created. If it had been created that route pointer was 
used. If the route did not exist it was created and added to the state variable array that 
contained route pointers for that node. This way the route pointer would be available for 
later use and no unnecessary routes would have to be built. 


c. Xmt_pkt State 
The processing of commands in OPNET Modeler adds no time to the 


simulation time. Time is only added by specific calls of the OPNET function 
“op_pk_send_delayed.” In reality for any node to process incoming frames takes finite 
time. This time or processing delay is simulated through use of the 
“op_pk_send_delayed” function. Current implementation uses an exponential PDF as to 
model these delays. However, with vendor data better models of this delay could be 
implemented, further enhancing the fidelity of the model. 


: THE SWITCH FINITE STATE MACHINE 
The switch in Fibre Channel is the heart of what has been referred to as the fabric 


configuration in Fibre Channel. In this model the switch was viewed in fairly simplistic 
terms. A switch’s job is first to receive frames from one node or switch. Next it must 
accomplish any administrative overhead associated with the protocol. For this Fibre 
Channel mode] these tasks are mostly limited to sending R_RDYs back to the sender in 
order to ensure flow control. Lastly, it must transmit the frame to the next switch on the 
route or to the final destination. For these reasons it was modeled as 32 pairs of 
transmitters and receivers configured around a processor. It is described as a 32-port 
switch. Like the node, the processor is modeled by a FSM. Like the node FSM the 
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switch FSM is broken up in to several areas that correspond roughly to their 
functionality. These are the Initialization Path, the Process Path, and the Transmit Path. 


The switch FSM is shown in Figure 12. Appendix C contains the OPNET code for the 


switch FSM. 
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Figure 12. Fibre Channel Switch Finite State Machine. 


1. Initialization Path 
Just as with the node several administrative tasks have to accomplished in the 


switch before it can process any network traffic. This accounts for procedures that would 
take place during an explicit or implicit login and contains five states, init, objid?, 
subnet?, con_out, and set_link_cost. 


a. Init State 
During the init state three functions are accomplished. First the Buffer-to- 


Buffer credits are set to the login value. Second, utilization for the links is set to zero. 


Utilization is used in the routing of frames. In order to ensure that the bandwidth across 
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all redundant links 1s balanced, frames are sent on the route with the lowest utilization. 
Lastly, global statistics are initialized and the first interrupt used to set link cost is 
scheduled. 


b. Objid? State 
The objid? state in the switch model functions almost exactly as the 


objid? state in the node model. The switch is given a unique ID and the values for mean 
processing delay and mean R_RDY delay are set. In an abstract sense a switch simply 
routes frames around the fabric. This routing takes finite time. The two delays can be 
thought of in the following manner. The mean time it takes to recognize that the 
incoming signal is a frame plus the time required to construct and send the R_RDY 
primitive back to the previous node/switch equals the mean R_LRDY delay. The mean 
time it takes to analyze that frame and determine how big it is where it 1s going, find out 
if that resource is available (through flow control), how to get it there, and finally to send 
it is what is defined in this model as the Processing Delay. 


c. Subnet? State 
In order to route in OPNET both an objid and subnet are required. The 


subnet? state is used to determine the subnet of the node. This is an administrative state 
only and is only required so that the OPNET Modeler routing functions may be used. It 
neither adds nor subtracts from the fidelity of the simulation. 


d, Con_out State 
The state con_out is designed to find all links attached to the switch and 


build a look-up table, so that when a packet is routed to a node/switch the transmitter 
attached to that node/switch can be found quickly. This state has no impact on the 


fidelity of the simulation. It does, however, affect the speed and resources required in 
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running the simulation. The OPNET functions required to build the arrays of attached 
Fibre Channel link models, nodes and switches are very expensive computationally. If 
these functions had to be called each time a frame was forwarded, simulation run time 
would become inordinately long, rendering the simulation all but useless. 


é. Set_link_cost State 
In order to ensure balanced bandwidth utilization across redundant links, 


cost of traversing a link had to be a function of bandwidth utilization. The set_link_cost 
state is an asynchronous state (much like the topo_build state in the node FSM) that 
computes the utilization of each link and applies it on a scheduled basis. The more often 
this state 1s called the more accurate the simulation. The price paid is in simulation speed 
and memory resources. 


2. Process Path 
The Process Path is that portion of the switch FSM that looks at the received 


packet (primitive or frame), analyzes it, and decides what actions must be taken in 
response to that packet. This path contains the following states, idle, type, r_rdy, frame, 
and dec_ buf. 


a. Idle State 
The idle state could easily be thought of in either the process or transmit 


path. It is simply a state from which the process waits to be invoked. The idle state 
represents a switch waiting to receive or forward traffic. 


b. Type State 
This state assesses the type of packet received at the switch receiver. To 


do this task the state counts the number of fields in the packet received. For any 
primitive there are less than three fields, whereas for a Fibre Channel frame there are 


many more. 


a 


c R_rdy State 
In this model, if a packet received has less than three fields it has to be a 


Fibre Channel primitive. Since only R_LRDY primitives are supported in this model it has 
to be an R_LRDY. Upon receipt of an R_RDY this state determines which node sent the 
R_RDY and increments the buffer-to-buffer credit for that node. Since the credit amount 
can never be greater than the initialized credit value, a mechanism was put in place to 
prevent credit from exceeding that value. 


d. Frame State 
Each packet transmitted in the OPNET simulation has an associated route 


pointer. This pointer references the route that the packet will take from its source to its 
final destination. If the switch receives a frame the frame state advances the route 
pointer so that the frame is directed to the next node in the route. 


é. Dec_buf State 
When a frame reaches the dec_buf state some determination must be 


made as to the status of the switch’s buffer-to-buffer credit with the next node in the 
route. The dec_buf state checks to see if buffer-to-buffer credit is available, and then 
takes actions based on that information. 

If the frame is Class 3 and no credit is available with the next node the 
frame is destroyed and the FSM transitions back to the idle state and awaits the arnval of 
the next packet. The additional step of incrementing the “Dead Class 3” Statistic is 
completed here to aid the system designer in system optimization. If the frame is Class 2 
and no credit is available with the next node the FSM transitions to the f_bsy state. 


f. F_bsy State 
If a Class 2 frame (either FT-1 Data frames or FT-O Link Control Frames 


(LCF)) cannot be forwarded to the next node in the route, some notification must be 
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made to the originator of the frame. The f_bsy state accomplishes this goal by turning 
the frame into an F_BSY LCF and returning it to the onginator of the frame. 


a Transmit Path 
The Transmit Path in the switch consists of three states, add_route, sndrrdy, and 


xme¢. 


a. Add_route State 
When there is not enough buffer-to-buffer credit available to forward a 


packet and an F_BSY must be returned to the originator of the message, the switch is 
forced to add a route to the F_BSY from the switch to the onginator. The add_route 
state computes a minimum cost route from the switch to the onginator of the frame and 
adds that route pointer to the packet. 


b. Sndrrdy State 
The sndrrdy state constructs, routes, and sends with some user defined 


delay the R_RDY in response to the frame received. Current implementation allows for 
constant delays in the switch, however PDF based delays could easily be added to 
increase simulation fidelity. 


C. Xmt State 
The xmt state serves two functions. First it adjusts the utilization of the 


link which the packet is being sent out on. Since the routing in this switch uses a 
minimum cost algorithm to route packets, and cost is a direct function of utilization, this 
State updates the utilization for each packet sent out of the switch. Second, this state adds 
any delay associated with the processing of the frame. In the current implementation this 
is a constant, but for better fidelity it could be modeled by a PDF using vendor supplied 


data or test output. 
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V. MODEL VALIDATION 


A. OVERVIEW 


In order to ensure that models built with the simulation tool give accurate results 
some validation process must take place that ensures that the model represents what 
happens in the physical article. Several tests were conducted using the Fibre Channel 
model that was designed in OPNET Modeler in order to ensure that any conclusions 
drawn from the model could be considered accurate. 


B. TESTS 


1. Throughput 
The first test involved finding the functional relationship between the interarrival 


argument of the Packet Generator Object and the actual throughput of the link. By 
knowing this relationship actual objects can be modeled with various throughputs simply 
by changing the value of the Packet Generator Object’s interarrival argument. 


a. Theory 
Throughput will be defined in bits/sec. Interarrival argument defines how 


often a new packet (or frame in the case of Fibre Channel) is sent from the Packet 
Generator Object. Interarrival argument is represented in frames/sec. Using dimensional 


bits . frames _ bits 


analysis it can be shown that by multiplying Since in all 


frame sec Sec 
likelihood the system designer will have a target throughput of a node and will want to 
solve for the interarrival argument. In this case the format would be 


it 
BEC ceils =——— As an example if a frame is composed of 20,000 bits and the 
bit frame frame 


desired throughput is 5,000,000 bits/sec the interarrival argument would be 0.004. 


oy 


b. Model 





Processor 


Figure 13. Model Used to Verify Throughput and Overhead. 


The model to test the throughput was a simple connection of a processor 
node connected to an antenna node. This configuration is shown in Figure 13 above. 


c: Test and Test Conditions 
Since there were only two nodes in the system, all traffic was forced to the 


other node. This would ensure that all bandwidth utilized was a function of the node of 
interest. In Fibre Channel the longest frame possible is 21720 bits. The breakdown of 


one data frame of maximum length is broken down in Table 4. 


Table 4. Frame Size Breakdown 
Number of Bits Required for 






Transmission 


Frame Header 240 Bits 


End of Frame (EOF) 40 Bits 













Required Inter-Frame Bits 240 Bits 


Maximum Size Data Payload 21120 Bits 
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4.0msec was chosen as the interarnval argument. This equates to a throughput of 
approximately SMbytes/sec (5,430,000 Bits Per Second (BPS) to be exact). Since each 
frame sent will require the receiver to send an R_RDY in reply to all frames and in the 
case of Class 2 frames an ACK frame. This would corrupt the data, so to ensure that the 
data recorded was only due to one of the nodes only the antenna would transmit at the 
desired data rate. The processor would wait to transmit its first frame until after the time 
of interest. Also, only Class 3 frames were sent so that no ACK frames were created. 


d. Results 
The simulation was run for 10 seconds and the results are shown in 


Figure14 below. 
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Figure 14. Results of the Simple Throughput Validation Test. 


The chart clearly shows a throughput of 5,430,000BPS, which is exactly 
what was predicted. 


2. Overhead 
Some method was needed to verify that for every frame sent across the network 


the proper amount of return overhead was returned. That is to say that for any Class 
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frame a R_RDY was returned, and for Class 2 service not only was the R_RDY sent, but 
also an ACK was sent. 


a. Theory 
An R_RDY represents 40 bits. If only Class 3 messages are sent, and the 


receiving node does not originate any data traffic, then for each data frame received 40 
bits are returned. So, for Class 3 the formula used to verify overhead is exactly the same 


rames bits bits 
J . The 





as the formula used to verify throughput, and that is 
sec frame second 


only difference is that in the Class 3 example the return “Frame” is only 40 bits long. 

For Class 2 messages it was assumed that for every frame sent an ACK 
was returned as well as the R_RDY. This adds overhead on the return path. The basic 
formula remains the same, but to get the total BPS of overhead one must add the 
overhead due to the R_RDY and the overhead due to the ACK. When using the ACK_N 
method (described in chapter 14 of Ref. 4), which only returns one ACK for N data 
frames, the overhead will be reduced. This mode] does not support the ACK_N format. 


b. Model 
The same processor and antenna model used in the previous test (Figure 


13) was used in the Class 2 overhead test. 


é Test and Test Conditions 
The simple system shown in Figure 13 was configured so that only the 


antenna would originate data. This would ensure that all bits traveling on the return path 
from the processor would represent overhead traffic in response to the frame sent by the 
antenna. An interarrival argument of 0.004 seconds was again used giving a throughput 
from the Antenna to the Processor of 5,430,000 BPS. The test was conducted first using 


only Class 3 traffic and then using only Class 2 traffic. 
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d. Class 3 Traffic Results 
For Class 3 traffic the expected overhead is due only to the 40 bit RLRDY 


primitives being returned to the antenna for buffer-to-buffer flow control. The theoretical 


value would be 40/0.004 or 10,000 BPS. 
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Figure 15. Results of the Class 3 Overhead Verification Test. 


Figure 15 above shows that indeed the model accurately represents this. 


d. Class 2 Traffic Results 
For Class 2 traffic the only additional overhead is due to the ACK 


response to the message. Since the ACK is a 600-bit frame, the throughput would be 
equal to 600/0.004 or 150,000 BPS. This added to the 10,000 BPS caused by the R_RDY 
primitive results in a predicted throughput of 160,000 BPS. Figure 16 shows that the 
throughput on the return path from the processor to the antenna was exactly 160,000 BPS 


thus verifying the model’s behavior. 
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Figure 16. Results of the Class 2 Overhead Verification Test. 


3. End-to-End Delay 
The next test was conducted to verify that end-to-end delays inherent in Fibre 


Channel were modeled correctly. The tests involved two models. One was a long 
distance two-node network similar to the one used in the Throughput and Overhead tests. 
This model was used to verify that the propagation speed through the fiber was accurately 
modeled. The second was a was a short distance, two-node, two-switch network with 
aircraft like representative distances involved. 


a. Theory 
It takes time for bits to travel across a medium. Even in a vacuum nothing 


travels faster than 3.0E+08 m/sec. The time that elapses from the sending of the first bit 
to the reception of the last bit is the delay. Having predictable delays is the key to having 
a deterministic system. One analogy involves the train schedule. If the train arrives at 
the station at the same time every day it is easy to know when to be at the train station to 


catch the train. If, however, the train never leaves or arrives at the same time, it 1s very 
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difficult to make any decisions based on what amounts to a meaningless schedule. 
Knowing the time frame when frames will arrive after being sent is key to the system 
designer. 

The network modeled is a full speed Fibre Channel network using optical 
fiber as the medium. The full speed network operates at 1,062,500,000 BPS. At this rate 
it takes 0.94117647nsec to transmit one bit. At this rate a full data frame of 21,720 bits 
takes 20.44235usec to transmit. This delay represents the largest uncontrollable delay in 
the system. The only other uncontrollable factor is the geometry of the aircraft, which in 
turn drives the length of the interconnecting optical fiber. In a fighter/military aircraft 
wire runs of greater than 35 meters are unlikely. Given that, and assuming a propagation 
speed of 200,000,000 m/sec (based on an index of refraction of 1.5, and the speed of light 
as 300,000,000 m/sec) the maximum delay caused by the medium would be 0.175usec, or 
two orders of magnitude smaller than the transmission delay. All other delays are in 
some manner controllable by the system designer. It was this factor that drove the 
requirement to verify that end-to-end delay was a function of transmission delay plus the 
delay caused by the medium. 

For the purposes of this simulation two types of end-to-end delay were 
considered. First, when a Class 3 message is sent there is no acknowledgement so the 
end-to-end delay is clearly the difference between the time when the first bit is 
transmitted and the time when the last bit is received. For Class 2 a distinction was 
made. Since all in this model all Class 2 frames require an ACK (ACK_N is not 


supported), the end-to-end delay was calculated to be the difference between the time 
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when the first bit 1s transmitted and the time when the last bit of the returning ACK is 
received. 


b. Long Distance Model 
The first model designed was used to ensure that the propagation speed of 


the medium was properly measured. In order to do this a network was set up that was a 
known distance, in this case 10,000 meters. Only two nodes were involved. The model is 


shown in Figure 17. 


Antl Positian: Procl Position: 
<=! er OOOn x = 17,000m 
y = 50,000m Total distance between Antl and Pracl: y = $0,000m 

10, G3Cm 
=) 
ANTL Procl 


Figure 17. Long Distance Model. 

c. Test and Test Conditions 

This configuration was tested in two modes the first was using only 
Class 3 messages the second using only Class 2 messages. In order to keep the data 
uncorrupted the antenna was configured to transmit 100 full length (21,720 bits) data 
frames/second while the processor was confined to only responding to the antennas 
traffic and generating none of its own traffic. Neither the processor nor the antenna 
added any delay to the system. While this is physically impossible, this constraint was 
added so that the end-to-end delay would only be a function of transmission delays and 
delays due to the propagation through the media. For the first test one packet would take 
20.4423usec to transmit and 5Qusec to propagate through the fiber. Therefore the 


expected end-to-end delay would be 70.4423usec. Figure 18 shows the results of the test. 
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Figure 18. Results of the Class 3 End-to-End Verification Test. 


For the second test all the frames that were sent were Class 2. For this test 
the end-to-end delay would be expected to be greater than for the Class 3 case. Formally 
the delay should be equal to two times the propagation delay (100.0usec) plus the 
transmission delay for one full length data frame (20.4423usec) plus the transmission 
delay for one ACK frame (0.5647lusec). The sum of these would be 121.007059usec. 
During the test the actual value was found to be 121.0447usec, or a difference of 
37.6412nsec. It was determined during the model validation that this delay was equal to 
the time to transmit 40 bits. The source of this delay was the time it took to transmit the 
R_RDY back to the antenna. 

On the surface the R_LRDY should have no effect on the end-to-end delay 
of a frame. Clearly this was not the case and further investigation was required. The key 


is that as tested the nodes have no inherent delay. This means that when a frame is 
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received the R_RDY is constructed and sent instantly. The ACK is also constructed and 
sent instantly. Since the code that is used to construct and send the R_RDY occurs 
before the code that constructs and sends the ACK, the R_RDY enters the queue first and 
the ACK must wait until the R_RDY is sent before it can begin transmission. Results of 


the test are shown in Figure 19 below. 
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Figure 19. Results of the Class 2 End-to-End Verification Test. 


The results of the long-distance test clearly show that the model accurately 
simulates both the propagation delay and the transmission delays inherent in a Fibre 


Channel network. 
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d. Short Distance Model 
The short distance model was used to verify that delays through the 


switches were modeled accurately. The model contained one antenna one processor and 


two switches connecting the nodes. The model is shown as Figure 20. 
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Figure 20. Short Distance End-to-End Test Verification Model 


é. Test and Test Conditions 
The short distance model was tested using both Class 2 and Class 3 


frames. First with Class 2 and then with Class 3. The expected results for Class 3 would 
be that the delay would be equal to three transmission delays (for the full-length data 
frames) plus the propagation delay for 60 meters. The approximate value for this would 
be 61.627usec. For Class 2 one would expect an additional delay of three times the delay 
for the 600 bit ACK plus the delay for the 40 bit R_RDY plus the propagation delay for 
the 60-meter return. This equates to an additional 2.032usec or 63.659usec for the round 
trip. 


f. Results 
The results of the simulation are contained in Figure 21 below. 
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Figure 21. Results of the Short Distance End-to-End Delay Verification Test. 


The graph shows that the switch behavior is exactly as predicted, and 1s 
suitable for estimating the end-to-end delay inherent in an avionics system. 


4. Load Balancing 
The last tests that were conducted were used to verify that the bandwidth between 


switches was shared equally across all links. As was mentioned earlier, the best way to 
ensure determinism is to have excess bandwidth. By sharing the load between switches 
one can ensure that all links have equivalent excess bandwidth. 


a. Theory 
As was stated in the description of the FSM in the node, frames were 


routed based on a path of least cost. This cost was defined to be the throughput on the 


link, so that frames would travel down the link with the lowest throughput. 
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b. Load Balance Model 
The model was constructed using one antenna one processor and two 


switches. Between the two switches four links were connected. The throughput of these 
links would be monitored to ensure that as simulation time increased the throughput of all 


the links would be equal. Figure 22 shows the model used for the test. 





Figure 22. Load Balance Verification Model 


C. Test and Test Conditions 
The test involved setting the interarrival argument for each of the antennas 


to 0.004 sec for an aggregate throughput of 21,720,000 BPS. The processor was left in a 
receive mode only, so that the only data on the links between Switch 1 and Switch 2 
would be as a result of the antennas interarrival argument. 

The expected result would be that the throughput of each of the four 
switch links would tend to 5,430,00 BPS and the link between Switch 2 and Proc 1 would 


be 21,720,000 BPS. 
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It was assumed that the time it took for the model to reach steady state 
would be a function of how often the cost of the links was computed. As was mentioned 
earlier the cost was a function of throughput, and in the switch FSM this cost was 
calculated applied to the appropriate link on a reoccurring basis. The parameter that 
controlled this was the Global Variable DEL_COST. Two conditions were tested. The 
first test used a DEL_ COST of 0.5 sec the second test used a DEL_COST of 0.1 sec. 
Both tests only used Class 3 data since for the path from Switch | to Switch 2 using Class 
2 data would differ only by the 10,000 BPS required to send the R_RDY frames. 


d, Results 
Both charts, Figures 23 and 24, show that the load is indeed balanced as 


the simulation time is increased. This behavior verifies that when multiple links are used 
to connect switches the load across each of them will tend to become equal. The second 
result comes from the time required to attain this balanced state. For the first case 
(Figure 23), where DEL_COST = 0.5 sec the load is balanced + 5% at approximately 
29.00 seconds. For the second case (Figure 24), where DEL_COST = 0.10 sec, the load 


is balanced + 5% at approximately 5.48 seconds. 
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Figure 23. Load Balance Verification Test Result for DEL_COST = 0.5 
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Figure 24. Load Balance Venfication Test Result for DEL_COST = 0.10 sec 


Upon closer inspection there seemed to be a linear relationship between 
DEL_COST and the settling time. To determine this a MATLAB function was generated 


to do a linear least-squares fit of the results of several runs of the load balance test. This 
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code is contained in Appendix B. Analysis of the data showed that there was a good 
linear —_ relationship. The equation used for the curve fit was 


Settling _Time = 57.1332 x DEL_COST —0.507. The data is shown in Table 5. The 


results of the data and the curve fit are shown in Figure 25. A log-log plot was used to 
emphasize the points at low values of DEL_COST. 


Table 5. Data From the Load Balance Verification Tests 
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Figure 25. Relationship Between Settling Time and DEL_COST 


é. Conclusion 

From the perspective of Load Balance the model acts in a predictable way. 
While some designs may not want to ensure load balance, many will. The utility of the 
linear relationship between allows the system modeler/designer to predict the 
consequences of any value of DEL_COST before running the simulation. 


C. SUMMARY 


As designed the model simulates well many of the characteristics of a Fibre 
Channel system. The designer/tester of any Fibre Channel system should be aware of the 


limitations of the model. Future improvements to the model would include modeling of 
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the Session and Exchange layers. Better modeling of behaviors such as ACK_N, error 


tolerance etc. 
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VI. DESIGN, CONSTRUCTION AND TEST OF POSSIBLE 
AIRCRAFT ARCHITECTURES 


A. INTRODUCTION 


The purpose for designing this model was first to allow a systems engineer to 
propose a design for an avionics system and test it under various conditions, and second 
to take an existing system or component and ensure that it meets the design 
specifications. During the design phase it was assumed that no components had been 
specified other than that they had to support Fibre Channel. For the validation phase it 
was assumed that all components represented actual hardware and could not be changed. 
This section will attempt to show the design and test of simple systems. Some of the key 
areas that will be investigated include measuring end-to-end delays, computing 
appropriate credit values, testing the effect of Class 1 traffic and viewing the effects of 
system failures. End-to-end delays relate to determinism. Credit requirements relate to 
the amount of memory required by the different components. Lastly the effects of 
Class 1 traffic and system failures will show the robustness of the system. 


B. DESIGN OF A SIMPLE SYTEM 


In order to demonstrate some of the capabilities of the model a simple four node, 
one switch system was constructed. The system has one radar antenna, one radar 
processor, one display and one display processor. A single 32-port switch connects these 
components. The system is designed so that the radar antenna and radar processor can 
communicate with each other, with the radar processor having the additional capability of 
talking to the display processor. Only the display processor can communicate with the 


display. The model is shown in Figure 26. 
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Figure 26. Simple Aircraft Avionics System 


fe System Parameters 
For this example it was assumed that the desired data throughput from each node 


was a design point. Table 6 shows the throughputs. 





Table 6. Desired Values for Node Through 


Throughput Out 


Since no hardware was specified the system would be designed using several 







values for delay. It was assumed that the time required to send an R_LRDY would be very 
quick so values of RLRDY Delay were set to 1.0nsec, and 100nsec. The time to forward 
packets through the switch (Switch attnbute, switch.Processing Delay) was defined for 


SOnsec and 5usec. Processing times for the various nodes (proc.Processing Delay) were 
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also set to 500nsec and S5Qusec. It is clear that even for this small network many 
combinations of delay times exist. For this example one test case was analyzed of the 
more than 256 possible combinations. The test matnx is shown in Table 8. 


Table 7. R_RDY Delay = 100.0nsec, Switch Processing Delay = 500.Onsec 


Processing Delay 










The test condition would be run using Class 3 only, Class 2 only and a 50/50 mix 
of Class 2 and Class 3 messages. The goal of the system designer was to accomplish 
these throughputs with no lost frames, using as little RAM as possible. 


a Throughput Calculations 
As was stated earlier each node only broadcasts full-length data frames, ACK 


frames and R_RDY pnimitives. For the case of Class 3, outgoing throughput is 
approximately a function of interarrival argument. But, since the specification states that 
the data throughput one must also take into account the effect of 8b/10b encoding, the 
associated frame overhead, and the number of bits of data to be transmitted for each 
frame. For each bit of raw data 1.2 bits of Fibre Channel payload are generated. Also, as 
shown in Table 4, for each frame sent 600 bits of overhead are generated. Taking all this 
into account, for the condition that full data frames are sent (i.e. 2112 Bytes of data per 
frame transmitted) throughput on the link must be increased by approximately 28.6%. A 
MATLAB function was written to calculate interarrival argument based on these 
parameters, and is contained in Appendix D. In addition, there will be slightly more 


output due to the transmission of R_RDY primitives in response to Class 3 messages. 
di 


For Class 2 messages the output is even greater due to the requirement to transmit one 
ACK for each Class 2 frame received. Table 9 shows the interarrival arguments 
calculated which should satisfy the throughput requirements. 

Table 8. Calculated Frame Interarrival Arguments 

Node Interarnival Argument 
(sec/frame) 

0.003223 
0.3300 


0.1650 
0.01611 


3. Buffer-to-Buffer Credit Calculations 
The next step in the design of the system is to ensure that there is enough 







buffer-to- buffer credit in the nodes as well as in the switch. To do this the MATLAB 
function user_get_credit.m was wnitten. This code is contained in Appendix B. 


a. Node Buffer-to-Buffer Calculations 
Node buffer-to-buffer credit required is simply a function of how long it 


takes a frame to get from the node to the switch. This delay is a function of distance, 
frame-rate, link throughput, payload size, and number of nodes in the system, and switch 
delays. For a simple system it is fairly easy to analytically calculate these delays. These 
factors are fed into the user_get_credit.m function and values for delay are returned. 
These values can then be verified through simulation. Results from this showed that for 
small systems with small R_RRDY delays (less than 0.1 sec) buffer-to-buffer credit need 
not be greater than one. Table 10 shows the delays for the four nodes of interest. These 
delays are based on the sending of only full payload (21120 bits) Class 3 frames, on a full 
speed (1,062,500,000BPS) Fibre Channel] Link with an assumed propagation speed of 


200,000,000 meters/sec and a switch R_RDY delay of 100.Onsec. 
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Table 9. Time for A Node to receive R_RDY 
Distance | Propagation Frame 
to Switch Delay Transmission to Receive 
Delay R_RDY 


Antenna 
Processor 


Tae Osage | Sea ce 


Display 10m 20.442usesc ST Osisee 20.68usec 
Processor 


This chart assumes that the frames: expenence no other delays, and by the design 






Total Time 












all delays are constant and therefore completely deterministic. In general one can expect 
that not only will other delays occur, but also that the delays will have some sort of non- 
constant distribution. These delays will be small for small systems, but as complexity 1s 
increased they cannot be completely discounted. 


b. Switch Buffer-to-Buffer Credit Calculations 
The calculation required to determine the appropriate amount of credit for 


the switch is analogous to that required when determining buffer-to-buffer credit to the 
switch. There is however one minor difference. The switch must have adequate credit so 
that when more that one node is communicating with another node credit is available. In 
this example both the Antenna and the Display Processor communicate with the Radar 
Processor. Over time it can be expected that both will want to communicate 
simultaneously with the Radar Processor and hence the switch must have at least two 
buffer-to-buffer credits available for use. This formulation gets increasingly complex as 
multiple nodes are added and throughput rates are increased. Both factors must be 


considered in the determination. By allowing the system designer to try multiple values 
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quickly, this simulation tool will greatly reduce the workload and time required to 
evaluate/design an avionics system. 


4. End-to-End Credit Calculations 
For systems using both Class 2 and Class 3 frames the designer must ensure that 


enough End-to-End credit 1s available in each node. The steps required to do this are 
somewhat more abstract that those required for Class 3 only use. 

The first step is that the model must be constructed with adequate buffer-to-buffer 
credit to ensure that no frames are lost. The designer then runs the system for an 
adequate period of time with all Class 2 frames and evaluates the average end-to-end 
delay. From this data the user can build a histogram showing the possible delays and the 


frequency of their occurrence. A histogram for this sample system is shown in Figure 27 
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Figure 27. A Histogram Showing the Distnbution of End-to-End Delays 
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It is clear from this data that the delay is never worse than approximately 65usec. 
In fact upon analysis of the data 99.95% of all delays are less than 65usec. In order to 
ensure that frames are never lost the worse case delay must be used as one input to the 
checkCredit.m MATLAB function. The other input is the interarrival argument for the 
node of interest. For this test the longest delay of the system was 65.56usec. Table 11 
shows the end-to-end credit calculations for the system. 

Table 10. End-to-End Credit Calculations 
NODE Interarrival End-to-End Delay | End-to-End Credit 
Argument Required 

223.0usec/frame | _ 65.56usec 
165.0msec/frame | _65.56usec 


330.,0mseciframe | _ 65.56usec 
16.11mseclirame | _ 65.56usec 


Two main factors in this network would affect the number of end-to-end credits 












required. First, if end-to-end delays were to increase by several orders of magnitude each 
node would require more end-to-end credits to be allocated. For example if the longest 
delay was greater than 223usec two credits would be required by the Radar Antenna for 
communications with the Radar Processor. The second way more credits would be 
needed is by increasing throughput. This increase in throughput would decrease the 
interarrival argument. If the interarrival argument was decreased below the end-to-end 


delay value, more credit would be required. 


For this simple example the credits required could be calculated by hand. 
However, as the complexity of the system increases some kind of simulation will be 
required in order to gain insight into the delays and required credits. 
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5: System Test 
Once all parameters are determined for the system is run to ensure proper 


behavior. This simple system was run for one minute using a 50/50 mix of Class 2 and 
Class 3 frames. Full size data payloads were used and throughput was maintained at 
constant values. Delays for both the switch and all nodes were a constant 500nsec 
processing delay, and 1OOnsec R_RDY delay. 


a. Buffer-to-Buffer Credit Check 
The first data that was examined was to determine if any frames had been 


lost due to lack of credit at the switch. The statistics that would indicate this are the 
global statistics “Dead Class 3,” “F_BSY LCF,” and “F_BSY DAT.” All of these 
Statistics registered zero throughout the simulation. This indicated that the switch 
buffer-to-buffer value of two was sufficient. 

The second statistic that would show inadequate buffer-to-buffer credit 
would be the node statistic “BB Credit Reset.” Figure 28 below shows that for the 
Display, Display Processor and Radar Processor there was insufficient buffer-to-buffer 


credit for the given loading. 
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Figure 28. Buffer-to-Buffer Credit Resets For the Full System Test 


Clearly the graph shows that there is inadequate buffer-to-buffer credit in 
the Display, the Display Processor, and the Radar Processor. To correct this deficiency 
the values for buffer-to-buffer credit were incremented to two and the simulation was 
re-run. The second run indicated that sufficient buffer-to-buffer credit was available at 
all the nodes. 


b. Throughput 
The system’s throughput was evaluated using the OPNET tool. Graphs 


for the four nodes are shown in Figure 29. 
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Figure 29. Throughput From All Nodes, 50/50 Mix of Class 2 and 3 Traffic 


This graph shows the increase in data throughput due to the use of Fibre 
Channel as the communications protocol. Table 12 shows the average throughput of the ~ 
four nodes and the percent increase of throughput due to Fibre Channel Overhead. 


Table 11. Increase in Throughput Due to Fibre Channel Protocol 


Node Desired Data Actual Throughput Percent Increase 
Throughput 


100.0KBPS 











It is clear that the percent increase is highest for the nodes with the lowest 
throughput. This is due to the fact that those two nodes are also the recipients of frames 
from the highest throughput devices, the Radar Antenna and the Display Processor. 


Since 50% of the frames are Class 2, the node is required to send many ACK frames in 
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response to the input traffic. This bandwidth utilization passes only administrative traffic 
with no data payload, and 1s therefore overhead caused by the selection of Fibre Channel 
as the communications protocol. For this small system this fact may seem obvious, but 
the more complex the system the more difficult the traffic pattern 1s to analyze, and the 
more valuable a simulation tool becomes. 


c. End-to-End Delays 
Determinism 1s critically important in any avionics architecture. The 


measure of the range of times required for one frame to get from one node to another as 
well as the time required to receive the ACK that guarantees delivery of a frame is one 
way to describe how deterministic a system 1s. 

Delays in this model were broken up into Class 3 one way delays and 
Class 2 end-to-end delays. For the case of Class 3, since there is no guarantee of 
delivery, the delay was measured from the time the frame was transmitted until it was 
received successfully at the destination node. This is a one way time delay. For Class 2 
the decision was made to measure the time required to receive the acknowledgement that 
the frame was received from the destination node. This measurement will give the 
system designer some idea how long he should expect to wait before he can assume that a 
frame was somehow lost in transmission. In Fibre Channel this value is called the error 
detect timeout value or E_D_TOV. E_D_TOV is a cnitical system parameter that can be 
inferred/venfied through use of this simulation tool. 

Since all the delays in this system are set as constants there 1s almost no 
scatter and it 1S easy to determine the minimum and maximum delays for the system. 


Table 13 shows some of the statistics that were gathered from the simulation. 
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Table 12. End-to-End Statistics 
Max Delay Min Delay | Percent of Frames | Percent of Frames 
at Max Dela at Min Delay 
6.46565E-05 | 4.42141E-05 0.10% 99.80% 
6.24271E-05 | 4.19847E-05 99.80% 






This data clearly shows that the designer can expect greater than 99% of his 
frames to arrive at the minimum delay time and less than one percent of the frames to 
arrive around the maximum delay time. While for this system the histogram does not 
show any valuable information, in a complex scenario the determinism would not be 
clear and multiple simulations would have to be run. This OPNET tool gives the 


designer that capability. Figure 30 below shows the delays in graphical form. 
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Figure 30. End-to-End Delays for the Example System 
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VU. CONCLUSION 


Simulation 1s recommended to verify analytical calculations even for a simple, 
deterministic communications system such as MIL-STD-1553. For a more complex or 
non-deterministic system, simulation serves as a method of experimenting with new 
designs prior to implementation. Fibre Channel can be configured in a variety of 
systems, from simple point to point systems that are completely deterministic, to multi- 
hop switched fabric systems that are statistical in nature. This thesis has examined Fibre 
Channel as a potential avionics interconnection standard and described the development 
of a simulation model that allows detailed simulation of any Fibre Channel system. The 
model should be of great utility in the design and analysis of systems using the Fibre 
Channel standard. 


A. WHAT THE SIMULATION SHOWED 


Even though the simulation did not address many of the nuances of the Fibre 
Channel Standard it still gave some good insight to the performance one might expect 
from an avionics system using Fibre Channel as the interconnection system. 


1. Determinism and Latency 
As the simple simulation showed, even with constant delays at the switches and 


with only four transmitting nodes, total delays varied from between 42sec to 64sec. 
This performance gets less deterministic as network loading increases and more nodes are 
placed on the system. Increases in network traffic result in more packets queuing up at 
each switch or node. This makes it harder to determine exactly when a given packet will 
arrive at its destination. Clearly, one of the biggest factors in the latency of the packet is 


how many switches it must transverse. For each switch that is crossed a full size data 
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packet will pick up 0.94nsec of delay per bit transmitted. This is in addition to any 
routing delays encountered. For a full size data frame this delay is 20.44 usec. 


Zs Problems With Displays 
One of the areas of concern when using Fibre Channel is the amount of bandwidth 


required to support high-resolution displays. To illustrate this point assume that the 
display is a 10 inch by 10-inch display with a resolution of 1024 pixels by 1024 pixels. 
This equates to 1,048,576 pixels per display. If 24 bits per pixel are used and the screen 
is refreshed at a 30 Hz rate, the resultant bandwidth is 720Mbits/sec. If Class 3 data 
frames are used to send this traffic and no other traffic is on this link, the required 
throughput is approximately 926Mbits/sec. This represents a bandwidth utilization of 
over 90% on a full speed Fibre Channel link. 

Clearly, to support modern day displays one of three solutions needs to be 
imposed. First, some of the processing burden of the node of interest could be shifted to 
the display itself. This is the concept of the “Smart Display.” Second, some sort of data 
compression could be used. Lastly, a dual speed (2 Gbits/sec) or quad speed (4 
Gbits/sec) could be used for displays and their associated processors. 


3. Burden of Administrative Overhead 
Systems with increasing complexity and an increasing need for determinism will 


need additional administrative packets active on the network. These packets will reduce 
the amount of bandwidth available for actual data. Examples of this sort of traffic 
include arbitration packets that will determine which packets have priority in a network 
as well as routing packets that determine optimal routing between various nodes. Also, 
the choice of transmission class can cause differing quantities of acknowledgement 
packets to be returned. If the return channel is already heavily loaded, too many 
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acknowledgement packets may overload the system. As these packets are returned over a 
different channel than the data they are acknowledging, it is easy to overlook their impact 
on the system. Clearly, fine grain analysis needs to be done to determine the effect of 
this overhead on overall system performance and bandwidth utilization. Stating this 
another way, one cannot assume that just because a node only has an output of 
600Mbits/sec that it can be inserted in a 1 Gbits/sec network and expect it to have no 
impact on overall system performance. 


4, Difficulty In Analytical Estimates For Complex Systems 
Even in the small system built for this thesis, analytical analysis was difficult. 


When the network size is increased, the delays become not only less deterministic, but 
also functions of multiple variables. This will ensure that accurate analytical analysis 
will be impossible. 


B. WHY FIBRE CHANNEL CAN WORK 


Fibre Channel is gaining momentum in the commercial industry. Large amounts 
of research are being conducted to improve the standard. As this continues it Is 
increasingly likely that more and more applications will be found for its use, and cost will 
continue to decrease. There is an active working group (FC-AE) addressing the specific 
needs of avionics architectures. Currently, it is being used in such programs as the F-18, 
B-1B and AWACS aircraft [Ref. 14]. The lessons learned from these implementations 


will help ensure success of future programs. 


Fibre Channel meets all the technical requirements of an avionics interconnect. 
Specifically, it is a deterministic, real time serial bus capable of sending many types of 
data between microprocessors. It is scalable, plug-and-play, hot swappable, and easy to 


maintain. Table 14 shows how well the Fibre Channel switched fabric topology meets 
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the goals of the JAST Mission Systems IPT. Further details concerning the use of Fibre 
Channel in an avionics system are contained in Chapter II of this thesis. 


Table 13. Digital Network Requirements. After Ref. [ 1 J. 















Requirement Third Fourth Generation | Fibre Channel Switched 
Generation Fabric 


Network Size <32 connects <256 connects 


Data Rate-per path 


Quad Speed 4 Gbps 
Data Rate-aggregate 









There appears to be no technical reasons that Fibre Channel could not replace the 
current avionics backbone: MIL-STD-1553B. 


C. WHY FIBRE CHANNEL MAY NOT WORK 


The majority of the work in Fibre Channel has been for storage area networks. At 
a recent three-day industry exposition (FCTC 99, San Jose, CA) of Fibre Channel 
products and presentations, there were no companies noted that were currently working 


on any military applications. 


The standard is weak and has loopholes. There have been many instances of two 
devices that conform to the standard, and yet when connected together the system will 
not work as a whole. In his keynote address at FCTC 99, Richard Lary explains that, due 


to the many possible interpretations of the standard, it is likely that any two devices from 
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different companies will not work together. This fact will severely hinder the design of 


truly open systems architectures. 


Lastly, while the interface to a fabric switch conforms to a standard, the insides of 
the switch are proprietary. This may make it difficult to second source the switch, further 
limiting the choices for other commercial alternatives. 


D. FOLLOW ON WORK 


The following paragraphs address the shortcomings of this model. 


1. Session and Exchange Layers 
This thesis only models the physical and link layers of Fibre Channel. Important 


additional insight to Fibre Channel’s performance could be obtained if the model’s 
fidelity was increased to include the session and exchange layers. The addition of these 
layers in the simulation would quantify the expected increased bandwidth utilization and 
decreased determinism that would come as a result of the use of sessions and exchanges. 


pa Probability Density Function Models of Real Systems 
During the testing and simulation of the simple avionics architectures there was 


no real reason for the choice of constant delays. In reality the delays would follow some 
sort of PDF. This PDF would have to be obtained using either vendor data or a Monte 
Carlo simulation based on some assumptions about the switches and various nodes. The 
subsequent use of this PDF would add fidelity to the model. 


3. Comparison of Simulation Data to Real System 
Ultimately, the quality of a simulation is determined when the results of the 


simulation compare well to actual data obtained from system tests. Unfortunately, at the 
time of this writing no data was available for comparison. Future research in this area 
should try to obtain throughput, overhead and end-to-end delay data from an actual 


system and compare that data to data generated by the simulation. 
ZH) 
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APPENDIX A. COMPARISONS OF OTHER INTERCONNECT | 
SYSTEMS 


Ie OVERVIEW 


This purpose of this thesis is to show how well Fibre Channel suits the next 
generation avionics interconnect. The following are some of the competing technologies 


and why they may or may not be a suitable alternative. 


a. Firewire 

Firewire or IEEE-1394 as it is also known is a fast (LOOMbit/sec to 400Mbit/sec), 
peer-to-peer serial bus. It supports hot swapping of devices and can be used to transport 
a large variety of data. It is working to become the standard interface of the desktop 


computer. 


Firewire has several limitations. First, the network is in the form of a tree 
topology with limited distance between devices. Second, since it’s market appears to be 
limited to the desktop market, very few ruggedized components are commercially 
available. [Ref. 14] It is unlikely that IEEE 1394 will gain a large share of the military 
interconnect market, and hence a somewhat risky technology upon which to base future 


avionics architectures. 


b. Gigabit Ethernet 
Since it lives in the collision domain, Ethernet 1s inherently non-deterministic. 


This fact alone makes it unsuitable for an avionics application. 


8: 


e Asynchronous Transfer Mode 


Asynchronous Transfer Mode (ATM) 1s a fast (622 Mbits/sec) {Ref. 15] serial bus 


designed for the telecommunications industry. Primarily designed for voice/video, it’s 
disadvantages include that it is a lossy protocol, has high software overhead and that it 
has had many delays in both hardware and software standardization. [Ref. 14] (It is 
interesting to note that ATM protocol can be run over a Fibre Channel network.) Also, 
there does not appear to be a preponderance of ruggedized components for ATM, which 
are necessary for use in a military combat aircraft. ATM does not appear suitable for use 


in avionics architectures at this time. 
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APPENDIX B. OPNET NODE FINITE STATE MACHINE CODE 
1. LIST OF STATE VARIABLES 


Table B-1. List of State Vanables Used in the Node Finite State Machine 


Variable Name Variable Type States where variable is used 


sig | im | objic2, FC_1, b_plate, route, snd_rrdy 
| did tist?200) | Objid___|_get_pkt, FC_0, FC_1, others, computers _ 
subnet_list[200] set_pkt, route, others, computers 


Pond see 


proc_delay objid?, xmt_pkt 


r_tdy_dela objid?, snd_rrd 


| objid? sndrrdy 
int 
Stathandle 
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2. LIST OF TEMPORARY VARIABLES 


Table B-2. List of Temporary Variables Used in the Node Finite State Machine 


Variable Name Variable Type States where variable is used 


node id | Objid objid?, b_plate 
node_type{10} 


node_name[ 10] Chan objid? 
subS ystem{10) 


subnet_id subnet 


subnet 


total_ nodes computers, others 


temp_ids[200] computers, others 


imp_char{10] 


pkptr Packet* eet_pkt, rev, CLI_xmt, CL22xmirg 
CL3_xmt, FC_0, FC_1, xmt_pkt, baplatcs 
snd_rrdy, r_rdy_ 
(ee get_pkt, FC_1, b_plate, route 
get_pkt, FC_1, route 
wane. te eee 


classtype | int b_plate 





daasize | im b_plate 
rset_ ptr Route_Set* route 
tmp_ route 


< 


numFields ‘© 


im 
cSRrdy | it : 
fc_type int snd_rrd 
src_nde snd_rrd 
ptr snd_ird 
classT ype wae = 


Q | | = 
” < 


\O 
ON 


FC_1,FC_0 


rere: FC_1 
ae FC_0 










f_bs int FC_0 
FC_0 
src_id FC_0 


3. HEADER BLOCK 


The following code is contained in the header block of the node’s process model. 


/* packet stream definitions */ 

#define RCV_IN_STRM 0 //Packet which comes from the node’s receiver 
#define SRC_IN_STRM 1 //Packet which comes from the ideal source 
#define XMT_OUT_STRM 0 //Packet which is sent to the transmitter 


/* transition macros are used to transition between states*/ 
#define SRC_ARRVLI ( op_intrpt_type () == \ 


OPC_INTRPT_STRM && op_intrpt_strm () == SRC_IN_STRM ) 


#define RCV_ARRVLI ( op_intrpt_type () == 
OPC_INTRPT_STRM && op_intrpt_strm () == RCV_IN_STRM ) 


#define REROUTE (op_intrpt_type() == OPC_INTRPT_SELF) 


#define COMPUTERS ( to_computers == OPC_TRUE ) 


#define OTHERS ( to_others == OPC_TRUE ) 


oe 


#define CLASS_1_XMT (class_type == 1) 
#define CLASS_2_XMT (class_type == 2) 


#define CLASS_3_XMT (class_type == 3) 


#define FC_O_RCV (fc_type == OxC) //Link Control Frame Recvd 


#define FC_1_RCV (fc_type >= 0x2 && fc_type <OxC) //Data Frame Recvd 


#define CLASS_3_REC (class_3_rec == 1) 


#define OTHER_CLASS_REC (class_3_rec == 0) 


#define RLRDY_RCV (c3Rrdy == 1) 


#define FRAME _REC (c3Rrdy == 0) //Either LCF or Data 


// Global Variables // 
// // 


| [ER ER A ER RR EK RR A RR RC RRC EE CAE ACE AC AC C2 EE 2 2 Eo 2 2 ak Ko aK 7 


const int MAX DATA _ SIZE = 21120; //Maximum Data Bits allowed in one Frame 


const int MIN_DATA_SIZE = 0; //Define MIN_DATA_SIZE to 0 bits 
Topology *TOPO_PTR; /[The only Topology Pointer 

const int MAX_HOPS = 5; //Max number of Hops a path can have 
double LAST_TOPO_UPDATE_TIME = -1; 

double FIRST_AWAKE = -1; // Make sure Statistics are only initialized once. 
const double DELx f= 2. // Time between topology builds 
int F_BSY_DAT; // Holds the F_BSY_DAT stats 

int Boy Ler: // Holds the F_LBSY_LCF stats 
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ta ac Fe tee ote a he hee he Ae Ae A aR a a wise EA OO SSE CE AE 2G 2 2 AG fe He He Ie AC HE ff 


// Function Prototypes // 
/ // 


Je See SSE pe ee Ses ast a Senor ie Sas ea Se coc Ae See ea Sr Ta Sc se cea a ec ea 
// Function: int data_sz_set(int) 


// Purpose: Returns data with size 0 - 21120 bits 


int data_sz_set(int); 


//end Header 


ea ee Fe Ne Ie Fee He 2 2c ok 2h ehh eee 2h Me eich ake Ae fee 2h A le hah ete a ra ee ee Ee / 


4. FUNCTION BLOCK 
The following code is contained in the function block. 


tae ete cha 2h he ke he of hs ots erie ete cbs fe 2 ete oF A 2h oh ie ole Ee oh kt 2 ee ee Ee re 


// Function: int data_sz_set(int) 
// Purpose: Returns data with size 0 - 21120 bits 
int data_sz_set(int dta_size) { 
int max_data_size; 
//max_data_size gives an exponential dist of data with mean of dta_size 
//dta_size in in words of data payload, while max_data_size is in bits 
max_data_size = 40*(int)op_dist_exponential(dta_size); 
if (max_data_size >= MAX _DATA_SIZE) { //1.€ greater than 21120 bits 
retun MAX DATA _ SIZE; 
\ else if (max_data_size < MIN_DATA_SIZE) { 
retumn MIN_DATA_SIZE; 
\ else™ 4 
return max_data_size; 
| Veudat 


\// end data_sz_set 


[PR AE ee ee ee ee 2g 2 2 2 2g 2 2 2 2 ee 2 a 2 a 2 2 2 2 2 ee 2 2A ee 2 fe ie fe 2 2 Ae fe 28 ke 2 fe 9 2 AC Ae ee Ae 2 2 2 he 2c 2 AE Ke 2 / 


oy 


5. INTE State 


The following is the code from the enter executives of the init state in the Fibre 


Channel node FSM. 


[ERROR ACR CR HOR RIOR ICRC HC ICR IC ACSI EK EK CK CRORE IC CFR IC IR KOR OR EKO ECE 
//declare global statistics (_gsh indicates a global statistic handle) 
ete_osh = op_stat_reg (ETE Delay’, 

OPC_STAT_INDEX_NONE, OPC_STAT_GLOBAL ); //for all packets 


ete_gsh_cl3 = op_stat_reg ("CL 3 ETE Delay’, 
OPC_STAT_INDEX_NONE, OPC_STAT_GLOBAL ); //only for class 3 


f_bsy_dat_gsh = op_stat_reg ("F_BSY DAT", 
OPC_STAT_INDEX_NONE, OPC_STAT_GLOBAL ); 


f_bsy_Icf_gsh = op_stat_reg ("F_BSY LCF", 
OPC_STAT_INDEX_NONE, OPC_STAT_GLOBAL ); 


//init the state variables for counting credit resets 
no_bb_count = 0; 


no_ee_count = 0; 

//register the local statistics 

no_bb_count_sh = op_stat_reg("BB Credit Reset", 
OPC_STAT_INDEX_NONE, OPC_STAT_LOCAL); 

no_ee_count_sh = op_stat_reg("EE Credit Reset", 
OPC_STAT_INDEX_NONE, OPC_STAT_LOCAL); 


// if youre the first one awake initialize the statistics variables. 
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if (FIRST_AWAKE < op_sim_time () ) { 
FIRST_AWAKE = op_sim_time (); 
F_BSY_DAT = 0; 
Rebs Y ECE 0; 


// load the class distribution (class_pdf is defined in the PDF editor) 
class_dist = op_dist_load ("class_pdf", 0.0, 0.0); 


//ititialize the buf_to_buf credit 
op_ima_obj_attr_get ( op_id_self (), "Buf to Buf", &buf_to_buf); 


max_buf = buf_to_buf; //can’t go higher than this, equivalent to login credit 


//schedule the first topology building interrupt 


op_intrpt_schedule_self ( op_sim_time (), 0); 


Fg ete cash ee Ae he che oF 2k Ac le otc ate ae fe AG 2 2 fe ake ake hc 2h oe ae Ie 6 2 Ae ots Ae of ety 2h 2 Sle cle ie he oes AC vn es a Ae =I Ae of oe Ao os AG RO oe AG so 


/ Global Variables // 
// // 
GE RR AA AR ate 45 OF Ae He AS 966 AS oe te iat Ae Ee ho A Aa Fert Ate et tee te ee ee ee ee J / 
/fint F_BSY_DAT: Holds data for F_BSY’s to data frames. 
//int F_BSY_LCF: Holds data for F_BSY’s to LCF frames. 
//double FIRST_AWAKE: Time the first node wakes up. 


Gh Ee AER Ae Ae He Fe A a rare Rat ih oe AE a telat a 


// State Vaniables // 
// I 
J [RAR ARR ARR A Ae He He A ee AE Oe 2 a AE OK fe 2 Ae 2 ie 2 6 2 ie 2A Oe 2 AS 2 2A 2 246 2 8 2 fe 2 2 fe 2 fe 2 eK a ae KE 2 i 2 2 2 EE EK 
//Distribution* class_dist: Holds the dist of the class of packets sent 
//Stathandle —ete_gsh: SH to the ete global stat. 
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//Stathandle 
//Stathandle 
//Stathandle 
//Stathandle 
//Stathandle 
//int 
//int 
//int 
//Ant 


ete_gsh_cl3: SH to the ete_cl3 global stat. 

f_bsy_dat_gsh: SH to the f_bsy to data stat. 

f_bsy_Icf_gsh: SH to the f_bsy to LCF stat. 
no_bb_count_sh: SH to the bb_count stat. 


no_ee_count_sh: SH to the ee_count stat. 


no_bb_count: Count of the number of times b_to_b 1s reset 
no_ee_count: Count of the number of times e_to_e is reset 
buf_to_buf: Buf_to_buf credits. 

max_buf: Login value of buf_to_buf credits 


] [PEPER IB EEE A OR 28 AS RO 2 OR A 2 0 EG 2s OB ae as oe le as Os is oP a AG ae 26 9 2K As Pe aie ae ots 26 he ie ats ie ots fe ok hs oe is inch oh 


// 
/| 


Temporary Variables // 
// 


J [RPE AR PR AEE Be Ae eA oe ee a Ae 2 ho ae 2 he Ae aR ols hy ee ake he ahs hs ha sats ele As che Ao op he ola oi he ais ats hs 2s As ee vip ee ots ots Hs nhs oko hs ota tra 


/| 


//end init 


NONE 


[5 RP A HA 2 2K KE A AE AC ER 2 A A C2 A A AC A EAC 2 2A 2 OC 2 a 2 2 2 EK OK / 


6. OBJID? STATE 


The following code is from the objid? state in the Fibre Channel node FSM. 


Yi a a ee ea ae a a es eS Se St a sa ci et Sade aS oa 


// This module finds out who the node is (name, type) and what 


//subsystem it is part of. Also the State variables proc_delay and 


//r_rdy_delay are set. 


node_id = op_topo_parent(op_id_self()); 


s_1d = node_id; 


//get attributes of the node model 
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op_ima_obj_attr_get (node_id, "model" , &node_type); 


op_ima_obj_attr_get (node_id, "name" , &node_name); 


//get attributes of the process model 


op_ima_obj_attr_get (op_id_self (), "Subsystem", &subSystem); 
Op_ima_obj_attr_get (op_id_self (), "Process Delay", &proc_delay); 


op_ima_obj_attr_get (op_id_self (), "R_RDY Delay", &r_rdy_delay); 


(CSCS ss Sodio St Seas chs toaea ae Sesto stime Stace Satta aceasta aia ig Se ec Se See eS Se er eS | 


// Global Variables // 


[CARH 2 A He HE He aH AE HO HE A A AE EH HE AC 2 EE FE EH AC 2 HC 2 2 CAC CEE 2 EC EE 2 2 IE 2 2 2 E22 2 2 2 aE 2 2 


/| NONE 


Ff eh obe Be Ke 2 ate Ae fe he Ac As ots He oh ole oi obs AS hs Ae te Ae 2 is hs Pa ata ect eA oR AS obs AG PR ic 0 iGo in nt oie oO os AS AS os ote She rig chs At oh RK A 


/| State Variables /| 


TSE SS Si a ase te eas Sr St a Se a Se acta Sa ec Sta Seat ee th Sos Se trent Staci Se Si Ses = aS Sa asl basta Mare a) | 


//int s_id: The source id of this node 
/[double proc_delay: The processing delay associated with this node 


/[double r_rdy_delay: The delay assoc with sending an r_rdy 


ia I ART i ote cla ofa ela ota athe ce chee Be oh eh Rate de ota Ae oh ie che te eee te aie oh ree oe ee ee ee ee 


// Temporary Variables // 
Fe ie ete Ra Ae is AS Te ots eI ATE le ots ota oh ts AG Se ete tert ote oh tail ta ote te oh otek ie ee cho oe i er re ee) 
//Objid node_id: The Objid of this node. 

//char* node_type[10]: The type of node (1.e. proc, disp, ant) 

//char* node_name[10]: The name of the node. 

//char* subSystem[10]: The subsystem of the node (EW, NAV, RDR etc.). 
//end objid? 


[FE AE AE EAE oe 2 EAE A A Ae Ae ae oie oR Ie eo AS As eRe 1S oe he oho oho he eR ote he acetate ele eink ofa ee ate he ote Ste obs oe hecho oe obs os 2 he He “AS io ote As AG 2 =: / 
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7. SUBNET? STATE 


The code for the subnet? state is contained below. 


subnet_id = op_topo_parent(node_id); 
subnet = subnet_id; 
//printf("My subnet # is: Yd \n", subnet); 


Pe RR Ea OR A re he AS SE A IE AS AC OR AS oR Ra ke oh oe OF ots he haat ote eo te ok let i et ee 


// Global Variables // 


PEEEX ER EEEEL EEE EEE EE EE EE ee ee Ee ee 


// NONE 


[EE AE Re Ee OF oe A OO Oe OR ek ae eee EE ee ee Ee ee 


// State Vanables // 


Baath th Re Ee Hh OF As ee eR oR oi oh He AB ah cts eel Pooh Ba ois oh Hee ol oho oie Fa ote hohe cin ees Fert te he tie cho rms ety oe 


//int subnet: The subnet of this node 
[PRR A EEA BE He EG EOE GE Oe OE 2 a A 6 as BO aS OF ss 2 Ae oO OS OE 2 OS 2 sR oo a RO oR Oe oe OE oe A i oo 


// Temporary Variables /| 


[PR AR Ae 2 2 2 OR ie Oe OS a he he he Ae Ae OE 2 2 he he AS ie 2h 2c 2g 24s 2K he 8 he 2c 2c 2c 2c 2c 2h fe 2c 2c 2g 2c 2c 26 2S 2 2 i he AE 2 Ds 24 24S 2 DS 2 ie Ac C2 OB 26 OK OE 2 OE  / 


//Objid subnet_id: The Objid of the subnet of this node. 


//end subnet? 


[FR FR RR OO Oi a Ae Ae 2 Ae Oe 2G Ae Ae A 2G 2G 2A 2G 2G AG AG 26 2S 2S OO OS OG 1G AE 2G he fe Ae fe 2h ie OF OF 2s Fe 2S 246 2 26 26 as Ae 26 8 2s 246 OF OO Oe 8 8 Ae Oe OF 2 OO 


8. FUNCTION? STATE 


The following code is contained in the function? state in the Fibre Channel node 
FSM. 


[PR AR 2K 2 2 Ae 2S Oe 218 2 he 2G 2G 2S OS 2 2G OE OK 2k fe 2A IG 2 fe 2 OE 2S fe 2h 2S EE 2s 2s ee 2S 2g 2 2 2 2S 9 Ke fe 2S 2s 2h 2S i 2 2 2k 2 2c 2c 2 246 26 288 2S 2 2 2 OE 


// This module finds out the function of the node inside the system. 
/{This information will help build the list of nodes that this node 


//will talk to. Each node has a type and a subSystem. 
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if (strcemp(node_type,"FC_ant")==0) { 
//printf("I am an antenna\n"); 


to_whom = |; 


\ else if (strcmp(node_type,"FC_proc")==0) { 
printf("I am a computer\n"); 


to_whom = 2; 


} else if (strcemp(node_type,"FC_disp")==0) 
//printf("I am a display\n"); 


to_whom = 3; 


\ else 
//printf("I am not working\n"); 


to_whom = 0; 
jaena ly 


if (strcmp(subS ystem,"EW")==0) { 
pnintf("I am in EW ss.\n"); 


what_subsys = 1; 


\ else if (strcmp(subS ystem,"EO")==0) { 
pnntf("I am in EO ss.\n"); 


what_subsys = 2; 
\ else if (stremp(subS ystem,"NAV")==0) { 


printf("I am in Nav ss.\n"); 
what_subsys = 3; 
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} else 1f (strcmp(subS ystem,"RDR")==0) { 
printf("I am in RDR ss.\n"); 


what_subsys = 4; 


} else if (strcemp(subS ystem,""DISP")==0) { 
printf("I am in Disp ss.\n"); 


what_subsys = 5; 


‘//end else if 


[FRE PEPE EAE AEE 7 7 EE 2 EO 2 AE ie AE Se 2 AS IC 28 6 7 ee ea i ee eee Ee Eee Ee 


// Global Variables HI 


JPET AER AR Oh Se I OE AG EH Ee AG AS AG ie ae ae 2 ake a 2 eae ee a He oe oi A 2 oe i Rr 2 OR oA ee ee ee 


// NONE 


J EEE A A A A AAG oe A 2h GTS Oh scrote Aaah is teint ote te Ae 2h aha 7 oleate Ste en erie cto he eta vr eloks Phy a ris oie ela Mee iG oie os ee 


// State Variables // 


[RP PR AE HH AR HH EH AR HE 2 HE AE 2 GA 2 A 2 2A C2 A HAC FAC AG 2A 2 A 2 AC OE 2 2 2 2 A 2 2 2 2 OK EK 


// NONE 

[FEE EE OG A EE EE AE Ee Ee OG 2 eo 2 AS OS 2 A I 2 ee oe oA Co ae oh ee 2 oe oe oo oo A eo 
// Temporary Variables // 
[FRE Ae RR OS A AG OE 2 AR OE OS 26 OR a Oo AE eo oI A a ot is 2 eo ee i ae AS ie 6 oe oe 2 is So a ae ote eRe oh 0B ole ots oie ots Pe 
//char* node_type[10]: The type of node (1.e. proc, disp, ant) 

//char* subSystem[10]: The subsystem of the node (EW, NAV, RDR etc.). 
//int to_whom: Used to specify who the node talks to 

//int what_subsys: What subsytem am | a part of 


//end function? 


[FR RB RAR BO Ae He He 2 Ae 2 He 2 AE 2 Ae 2 a He OIE ae OK he 9 2 eS 2 2 2 6 2 A fe 2 9 2 6 2 Ae ae 8 as Ae ae 2s a 2c 2 6 2 2 Ie OE AG 2K 2 OR OE OR oe 


2: I TALK TO STATE 


The following code ts contained in the i talk to state that 1s in the Fibre Channel 


node FSM. 


106 


[952 A HE A ACR AC EE EE ACR 2 2 EE CHE Hf A A 2 22 2 28 2 2 a a a a a ae a ae 


// This module aids the FSM in its transition to the building of the communications 
//lists. 


switch (to_whom) { 


case 0: { 


//printf("I talk to no-one\n"); 


break; 
j 
case 1: { 
//printf("I talk to computers\n"); 
to_computers = OPC_TRUE; 
break; 
} 
case 2: { 
//printf("I talk to displays, computers and antennas\n"); 
to_others = OPC_TRUE; 
break; 
} 
case 3. | 
//printf("I talk to display computers\n"); 
to_computers = OPC_TRUE; 
break; 
j 


\ //end switch 
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[PEAR FR AR AG ee Ae 2 2 GH os AIG AG GO HR iG hs is os Reha aka cole le ele cis ia ia ree i te ee 


// Global Variables // 
[PR PRR AE A Me AE 6 EA 2 BEE 2 a8 OE SAS i hi er OR ot A oA Ee RS nS oh 2 as A a 
// NONE 

[FEE AR M6 RAG Ah A Oe OS HG Oe AR OE He 2 he EO IE AS 26 EWG A AS eG ote Re Beats ate a chs ots whe Poin oho a ec A hae he ahr AG os hs ois Aa Oi Aes a ie ea 
// State Variables // 
[RPE PERG AE AEE AE DE AC 2 ee OE Ae eG A oe A nr Bi or Ae a a er 
// NONE 

[REE AE EE AE AE BE EE EE AC 26 AA eA ha he Fate EA Ach Si Re he ager acts tae ta 
/ Temporary Variables i 
[FEET i ACTH A 2 HG HS S26 EE AE A 2 Ge 6 2 OE OE AES 2 hs ie Ole nol 2 AG 2 AS ory 20 AC oe he SO EG Oe EO A 2 ee 2 
//nt to_whom: Used to specify who the node talks to 

//Boolean to_computers: True/False: I talk to computers? 


//end i talk to 


[AEA 2 oho Ae 2 Ee AS As 2h AS 26 2 a he ole Ac ais he 2s oe oh eta ais Ae ole Ao os mic 22h ee oh ih a ee 


10. COMPUTERS STATE 


The following code is contained in the computers state in the Fibre Channel node 


FSM. 


[BEA Ee 0 eee Ee Oe Ae 2 Oe oe Ae ae Ae he Ae A te os Oe oe ee Ae ote a ee oie ok 2 a AAS 2k oh le 2 Aa ols oP Ah 


//If you talk to computers the computers state will build arrays of the objects in the 


//simulation that you will talk to. 


[PRA AE 2 HO He Oe OR 2 Ee 2 Ae EO He ae OO AE AS Oe Ae EO oe He IG 2 AS 2 6 6 6 2S 2 2 Ae EO 8 8 OE 2s Ee 2 2 EO OE oe 


// count the nodes 


total_nodes = op_topo_object_count (OPC_OBJMTYPE_NODE); 


PRESET E SELLE EE EEE Ee ee ee ee eee ee eo eee ey 


// initalize the arrays, —99 is a bogus Objid 


for (i = 0; 1 < 200; i++) { 


subnet_listfi] = -99; 
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di siaehistii|iee— 09: 
e_to_ef[i] = 


//printf("the total number of nodes in the system is %ed \n",total_nodes); 


[RRR RR RRR TR AT TE ACR CACC EC ERC RC EE IAC EC CCI OE CRS AC IC HK / 


// This loads all the nodes into a temporary array 


for (i = 0; i< total_nodes; 1++) { 
temp_ids[i] = op_topo_object(OPC_OBJMTYPE_NODE, 1); 
} // end for 


[RR RH OT AH ACA HE ACC AACA ACA AC 2 CE AC AC EI A OE OE OE OE RE CK / 


switch (what_subsys) { 
ease 1} /TEW subsys 


j=0; = /Anitialize the counter } 


for (i = O; 1< total_nodes; 1++) { 
//get the model name 


op_ima_obj_attr_get (temp_ids[i], “model”, &tmp_char); 


//if it’s a computer 
if ( strcmp (tmp_char,"FC_proc" ) == 0 ) { 


op_ima_obj_attr_get (op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 


//and it’s an EW computer add it to the list 
if ( strcmp (subSystem, "EW") == 0 ) { 


op_ima_obj_attr_get(op_topo_child(temp_ids[i], 
OPC_OBJTYPE_PROC, 0), "End to End", &login_e_to_e[j]); 
e_to_e[j] = login_e_to_e[j]; 
subnet_list[j] = op_topo_parent(temp_ids[i]); //load the subnet 
d_id_list{j] = temp_ids[1]; //load the did 
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jt; /Ancrement j 


} 
\ //end if 
} //end for 
break; 
\//end case | 
case 2: { /TEO subsys 


j=0; = /Anitialize the counter j 


for i = O; 1< total_nodes; 1++) { 


//get the model name 


Oop_ima_obj_attr_get (temp_ids[1i], "model", &tmp_char); 


if ( stremp (tmp_char,"FC_proc" ) == 0 ) { //Af it’s a computer 


op_ima_obj_attr_get (op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subS ystem); 


//and it’s an EO computer add it to the list 
if ( strcmp (subSystem, "EO") == 0 ) { 


op_ima_obj_attr_get(op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "End to End", &login_e_to_ef[j]); 
e_to_e[j] = login_e_to_e[j]; 


subnet_list[j] = op_topo_parent(temp_ids{1]); //load the subnet 


d_id_list{j]} = temp_idsfi]; //load the did 
jt; //increment ] 
} //end if 
} /fend if 
} //end for 


break; 


\//fend case 2 
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€ase 3:1 INAV 
j=0; = //initialize the counter j 


for (1 = 0; 1< total_nodes; i++) { 


//get the model name 
op_ima_obj_attr_get (temp_ids[1], “model”, &tmp_char); 
if (strcmp (tmp_char,"FC_proc")==0) = { /Af it’s a computer 


op_ima_obj_attr_get (op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 


//and it’s a NAV computer add it to the list 
if (strcmp (subSystem, "NAV") == 0 ) { 
Oop_ima_obj_attr_get(op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), “End to End", &login_e_to_ef[}]); 


e_to_e[j] = login_e_to_e[}]; 


subnet_list[j] = op_topo_parent(temp_ids[1i]); //load the subnet 

d_id_list{j] = temp_ids[1]; //load the did 

jt 

} 
SS eongar 
} //end for 
break; 

\//end case 3 
case 4: { //RDR subsys 


y=0; = //initialize the counter j 


for (1 = 0; 1< total_nodes; 1++) { 
//get the model name 
op_ima_obj_attr_get(temp_ids[i], "model", &tmp_char); 


if ( strcmp (tmp_char,"FC_proc" ) == 0 ) { //Af it’s a computer 


el 


op_ima_obj_attr_get (op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 


//and it’s a RDR computer add it to the list 
if (strcmp (subSystem, "RDR") == 0 ) { 
op_ima_obj_attr_get(op_topo_child (temp_ids[1], 
OPC_OBJTYPE_PROC, 0), “End to End", &login_e_ texte: 


e_to_e[j] = login_e_to_e[j]; 


— ee 


subnet_list[j] = op_topo_parent(temp_ids[1]); //load the subnet 


d_id_list{j] = temp_ids[1]; /Noad the did 
jtt: 
} 
\ //fend if 
} //fend for 
break; 

\//end case 4 
case.s: { /[DISP subsys 


j=O; = /Amitialize the counter j 


for (1 = 0; 1< total_nodes; i++) { 
//get the model name 
op_ima_obj_attr_get(temp_ids[1], “model”, &tmp_char); 


if (stremp (tmp_char,"FC_proc" ) == 0 ) { /Af it’s acomputer 


op_ima_obj_attr_get (op_topo_child(temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 


//and it’s a DISP computer add it to the list 
if (strcmp (subSystem, "DISP") == 0 ) { 
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Op_ima_obj_attr_get(op_topo_child (temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "End to End", &login_e_to_e[j]); 


e_to_e[j] = login_e_to_e[j]; 


subnet_list[j] = op_topo_parent(temp_ids[i]); //load the subnet 


d_id_list{j]_ = temp_uidsf{i]; /Noad the did 
jtt; 
} 
1 /lend if 
\ //fend for 
break; 
\//end case 5 


\//end switch 


[5 RRR RR AH RR A ACHR EO A EAA ACR AC RR ACA RK EEE EE 2 2 EE 2 3 2 2 2 2 oa 2 oe 2 2 ok 2 2K / 
j=0; = /Anitialize j 
Te Ee Ee th St ome eS atk Satod Bo RRR Rae seat ho aR eae att RoR Me Mee sa a td aks ad 


//This just prints out the list of processers in (Sub_net id, d_id) form 


while (op_topo_parent(d_id_list[j]) !=OPC_OBJID_INVALID) { 
printf("(%d , %d) \n",subnet_list[j], d_id_list[j]); 
j++; 

\ /fend while 


num_proc = j; 
printf("Computers J is equal to %d \n",)}); 


//end computers enter execs 


[FEA AA A A AE EEE EE AE EA OE I EE AS AAG 2 AE OR AE Oe 2 2 Oe AE He A OS 2 OO IS Oe OS 8 2 He oo Hf 


// Global Variables // 


[RR A A AK AR RR EE EEE EE RR A CA EEE ECE ES RE a RC Co OE 2 IC ACK Ok / 


// NONE 
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[252 PRP A I EA 2 AR RR CC 2 RCC RA 2A 2A A 2 AC RC CIRCA CO a a 2 


// State Variables // 
[PRAIA AC Ba aia ahs Oh PIR A Ee 2 ES 2 AG AS PEG AG AC a Ie aR ata etacate os As hs AS 2 icici hac ae ah 2s A he 2s oh ols obs each os oh 
/Ant  d_id_list{200]: Destinations I might talk to 

/Aimt  subnet_list[200}]: Their assoc. subnets 

/int e_to_e[200]: End_to_end credit of the connected nodes. 

/Ant login_e_to_e: Login end_to_end credit 


[9578 RR RR A ROR RC ER I RR RR A IC CR CCR RII RC ICRC RCC CO 2 a a a aK aK 


// Temporary Variables // 
[PREECE AE 2 PE EE SR ERR anette oA AR aie eA a Ae Re he A Ae elo oto reer era 
//int total nodes: Total number of nodes. 

//Objid temp_ids[200]: Place to temp hold the Objids of the nodes 
//char* temp_char[10]: Place to temp hold the name of the node 

//char* subSystem[10]: The subsystem of the node (EW, NAV, RDR etc.). 
//int what_subsys: What subsytem am [| a part of 


//end computers 


[EEE ELELERES ERE EEE REE REE EE ee ee 


11. TOOTHERS STATE 


The following code is contained in the to others state in the Fibre Channel node 


FSM. 


//This state helps the processors decide which other nodes in the system they will 
//talk to and builds the communications lists. 


// count the nodes 

total_nodes = op_topo_object_count (OPC_OBJMTYPE_NODE); 
[EEFEEEELEETAELEE EAE ERATE EEE EEE EEE EELS OA 

// initalize the arrays to some bogus Objid value like -99 


for (i = 0; 1 < 200 ; 1++) { 
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subnet_list[i] = -99; 
d_id_list[1] = -99; 
e_to_e[i] = -99; 


[2 2 2 FRR A ACA AC EG 2 2A EC EE 2 2 2 2 2 2 2 AC CH 2 2 22K 


printf(‘the total number of nodes in the system is %d \n",total_nodes); 
// This loads all the nodes into a temporary array 
for (i = 0; i< total_nodes; 1++) { 
temp_ids[i] = op_topo_object(OPC_OBJMTYPE_NODE, 1); 
printf("‘temp_ids[%d] 1s Tod "1,temp_ids[i]); 
} // end for 
switch (what_subsys) { 
cases: { //If its in the EW subsys 
j=O; = /Amitialize the counter j 


for (i = 0; i< total_nodes; i++) { 


//get the model name 


op_ima_obj_attr_get (temp_ids[i], “model”, &tmp_char); 
if ( stremp (tmp_char,"FC_proc" ) == 0 || 


strcmp (tmp_char, "FC_ant" ) == 0 ) 
{ //if it’s a computer or ant/sensor 


Op_ima_obj_attr_get (op_topo_child(temp_ids[1J, 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 
if (strcmp (subSystem, "EW") == 0 || 
strcmp(subSystem, "DISP")==0) { 
//and it’s an EW computer, EW antenna, or display computer add it to the list 
Op_ima_obj_attr_get(op_topo_child(temp_ids[3], 
OPC_OBJTYPE_PROC, 0),"End to End", &login_e_to_ef[j]); 
e_to_e[j] = login_e_to_e[j]; 


subnet_list[j] = op_topo_parent(temp_ids[1]); //load the subnet 
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d_id_list{j] | =temp_idsf[i]: oad the did 


jt; 
} 
Pena it 
\ //end for 
break; 
\//end case 1 
Case. 274) /hf its part of the EO subsys 


j=0; = //initialize the counter j 
for (i = 0; i< total_nodes; i++) { 
//get the model name 


Op_ima_obj_attr_get (temp_uids[1], “model”, &tmp_char); 
if ( strcmp (tmp_char,"FC_proc" ) == 0 || 


strcmp (tmp_char, "FC_ant" ) == 0 ) { 
//if it’s a computer or ant/sensor 
Op_ima_obj_attr_get (op_topo_child (temp_uids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 
if ( stremp (subSystem, "EO") == 0 || 


strcmp (subSystem, "DISP") == 0 ) { 
//and it’s an EO computer, EO sensor 


//or DISP proc add it to the list. 
Oop_ima_obj_attr_get(op_topo_child (temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "End to End", 

&login_e_to_e[j]); 
e_to_e[j] = login_e_to_e[j]; 


subnet_list{j] = op_topo_parent(temp_ids[1]); 
d_id_list{j] © =temp_idsf[i]; //load the did 


jt: 
} 
\ /lend if 
} //end for 
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break; 


}//end case 2 


Case J} //If its part of the NAV subsys 
j=0; = //initialize the counter j 
for (1 = 0; i< total_nodes; i++) { 
//get the model name 


Op_ima_obj_attr_get (temp_ids[1], “model”, &tmp_char); 
if ( strcmp (tmp_char,"FC_proc” ) == 0 || 


strcmp (tmp_char, "FC_ant" ) == 0 ) { 
//if it’s a computer or ant/sensor 
op_ima_obj_attr_get (op_topo_child (temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 
if ( strcmp (subSystem, "NAV == 0 || 
Stremp(subSystem, "DISP")==0) { 
//and it’s an Nav computer, Nav sensor, or 


//display computer add it to the list 
Op_ima_obj_attr_get(op_topo_child (temp_ids[1], 
OFGSOBIMYTE PR@Gs))) End toend. 

&login_e_to_e[j]); 
e_to_e[j] = login_e_to_ef[}]; 


subnet_list[j] = op_topo_parent(temp_ids[1]); 
d_id_list{j] © =temp_ids[iJ; //load the did 


jo; 
} 
} /end if 
\ /fend for 
break; 
\//end case 3 
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case 4: { //RDR 
j=0; = //initialize the counter j 
for (1 = O; 1< total_nodes; i++) { 
//get the model name 


op_ima_obj_attr_get (temp_ids[1], "model", &tmp_char); 
if ( strcmp (tmp_char,"FC_proc" ) == 0 || 


strcmp (tmp_char, "FC_ant” ) ==0 ) { 


//if it’s a computer or ant/sensor 
Op_ima_obj_attr_get (op_topo_child (temp_ids[1], 


OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 
if (stremp (subSystem, "RDR") == 0 || 
stremp(subSystem, "DISP")==0) { 
//and it’s an RDR computer RDR ant, or 


//display computer add it to the list 
Op_ima_obj_attr_get(op_topo_child (temp_ids[i], 
OPC_OBJTYPE_PROC, 0), "End to End", &login_e_to_e[j]); 


e_to_e[j] = login_e_to_e[j]; 


subnet_list[j] = op_topo_parent(temp_ids[i]); 
d_id_list{j] =temp_idsfiJ; //load the did 


uses 


} 


\ //end if 
\ //end for 
break; 
\//end case 4 
case 5: { //DISP 


j=0; = //initialize the counter j 
for ( = O; 1< total_nodes; i++) { 
//get the model name 


op_ima_obj_attr_get (temp_ids[i], "model", &tmp_char); 
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if (Strempitmpmehnar, FC proc” )==0)|| 
strcmp (tmp_char, "FC_disp" ) == 0 ) { 


//if it’s a computer or display 
op_ima_obj_attr_get (op_topo_child (temp_ids[i], 


OPC_OBJTYPE_PROC, 0), "Subsystem", &subSystem); 
if (strcmp (subSystem, "DISP") == 0 ) { 
//if it’s a DISP computer or display 
op_ima_obj_attr_get(op_topo_child (temp_ids[1], 
OPC_OBJTYPE_PROC, 0), "End to End", 
&login_e_to_e[}]); 
e_to_e[j] = login_e_to_ef}]; 


subnet_list[j] = op_topo_parent(temp_idsj[1]); 


d_id_list[j] = temp_ids[i]; //load the did 
lias 
} 
} //end if 
\ //end for 
break; 
\//end case 5 


\//end switch 
[RAC CR ACE RC KC kK ICI ICR ER RE FC RE RR EAE FC ER a CR Ef ER a HA HE / 
j=0; = /Anitialize j 
[OR RRR AOR CR ICRA ICRC ICO EAC ICRC EC ICRA ICR EEC ACR 2 ICICI RE CCF RR EC ICE CR EK RE AC HCO / 
//This just prints out the list of processers in (Sub_net id, d_id) form 
while (op_topo_parent(d_id_list[j]) != OPC_OBJID_INVALID) { 
pnintf("(%d , %d) \n" subnet_list[j], d_id_list[j]); 
j++; 
} //end while 
num_proc = j; 


printf("Others J is equal to %d \n",}); 
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[#7 A ACH HH A A AC CC AR A RE 282 2 2 2 9 2 2A 2 2A 2K 2 2 2 EE 2 2 2k a 2 2K / 


// Global Variables // 
[PRE Ae A A a ER A Ee I ei ae ata aa te he te cto ce che oe Be he 2 2 os oh he hehe of he he ie oie a 2k ae 
// NONE 
[PR ARTE A AG He 2 A 2 ie He A A he 2 2 A 2 ee IG 2 Ae 2s 2 Ae oe ie ae oe 62 he ae 2h 2 2s 2k he 6 he 2 2 6 He 2 hee 26 2K ee 2 2 KK KK He KK 
// State Variables // 
[PRE RE Re eR A A He Ae 2 2h He Ae he Ee Ae 2 A 2 ae 2 he ee 2 Ae he 2 2s Ae AS AS Ae he i oe oe Se oe oe 2 2k Fe 2 ake ie fe fe 2 oe 2k 2 2k a 2 OK Oe OR a or 

/int  d_id_list{200]: Destinations I might talk to 

/int  subnet_list{200]: Their assoc. subnets 

/hint  e_to_e[200]: End_to_end credit of the connected nodes. 

/int  login_e_to_e: Login end_to_end credit 


J EEE CF Ae 2 a Ae ie ee AC Re ate ie oo le Fo ae Aa he oe oie EP are ie a Ae Ae at Re eS ie ke eal ee ee 


// Temporary Variables // 
[PEFR EE BE EE ee ee ee Ee Ae OR ei ee eee eek ke eee ee ee ee 
//int total_ nodes: Total number of nodes. 

//Objid temp_ids[200]: Place to temp hold the Objids of the nodes 
//char* temp_char[10]: Place to temp hold the name of the node 

fichat* subSystem[10]: The subsystem of the node (EW, NAV, RDR etc.). 
//Ant what_subsys: What subsytem am I a part of 


//end others 


[RE Ae EE OE Ee Ee Ae Reick 2h ee ae AE i Re ie ee ei ae Ae eer ere che cre oe hate cree eects eta hie ere 


12. TOPO_BUILD STATE 


The following code is contained in the topo_build state in the Fibre Channel node 


FSM. 


[EHR ARR RR RRR RH KR RE A A RR AC EC AAC ECAC ARCA A OE EK OK A CK / 


/[Topo_build updates the entire topology on a 
//scheduled basis set by global var DEL_T 


// reset the route pointers to null 


for (i = 0; 1 < 200; i++) { 
rte_arry[i] = NULL; 

} 

//Am I the first process to wake up? 

if (LAST_TOPO_UPDATE_TIME < op_sim_time()) { 
LAST_TOPO_UPDATE_TIME = op_sim_time(); //update Last update 
TOPO_PTR = op_rte_topo_from_objids(); //update topo_ptr 
op_intrpt_schedule_self(op_sim_time() + DEL_T, 0); 

} 


[SERS SS SR SRE ot Sia sec Ea Sa aS et ae ec cea cei sa eae to a tae ae crf 


// Global Variables // 


iste Ae ofa 2 AG 22h Ne =A 2 AS 2h 2h AG te As 2 he 2 2A he ho oA che he he he hac cr he oR ia As oe ect Ra a eee ef 


/[Topology* TOPO_PTR: The only Topology Pointer 
//double LAST_TOPO_UPDATE_TIME: When was the topology last updated 


Ge 2 he Ee AS OR OAS 2s As AS oe 5 OBS os A ae As oe hs chs he AR ots as fr che Ne ots a ots is ole eter ae ra ite ec a re a at eae ee Riche oe oh | 


/| State Variables /| 


SS Se See Se aa Se Se Su Se est eae Seco eae co a ic Seae ae See Cke Sece SeaeS SSeSe Se Feo eas ce Sea eat 
//Route* rte_arry[200}: Holds routes from this node to all others. 
gre teat cola es AS ee obs hn chs ob obe te ie che le oh cls eke ha ahs ce oh A Re 2 Rais Aa 2S is OR a i a a ar 


// Temporary Variables // 


[28 95 BR A A RR EA CE CAC A CE 2K KK EEE 2A RC ES A A ACK EE 2 oA 2 2 KC HE EK / 


// NONE 


// end topo_build 


[PR A 2 Ee 2 RH 6 2 2K ee 2 A Ae 2 EK Ae 2 AC He 2 Ee 2 AE HE 2 2 2 he 2 2 2 A 2 a a 2 2 2 2 2 A AC 6 ee 2 2 Ae 26 2 2 6 28 28 2 2 / 
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ES. GETPRK I STATE 


The following code is contained in the get_pkt state in the Fibre Channel node 


lasek 


[FE AR AE AG AEA is AE Ae 2 As Ae 2S AG AS 2 A AG 2 PE 8 2 RG 2g A 2s HE 2h He chs hs hs hs Ae a Aa ots Ae As ha oS 26 he 286 oe is 0 on is 2 Ae = 


//get_pkt is responsible for getting packets that 

//originated in the ideal packet generator source 

pkptr = op_pk_get (SRC_IN_STRM); 

[PRP AR AR FA 2 Ae A Hg 2 Ag 2 28 2 Hs AC A AR 2 Ee 2s AE AR AS 2s 2 2 Ae AS 2A 2s As 26 26 26 296 26 he 2K 2 26 2 ie 2c 2K fe 2c 2 OK He Ie OK ee 2 2 6 AE 2 2 eK A AG 
//Get an integer which is uniformly distributed 


//do while did = sid (don’t want to send to yourself) 


do { 
//uniform number cast as an int from zero to num proc which is | more than index 


dest_index = (int)op_dist_uniform (num_proc); 
printf("The index is %d \n",dest_index); 
dst_subnet = subnet_list[dest_index]; 
d_id = d_id_list{dest_index]; 
printf("My destination is (%d , %d) \n", dst_subnet, d_id); 


} while (d_id ==s_id); //make sure youTe not sending it to yourself. 


//end do/while 


[PR AR ARE FER A A AB AS Rs AS A AS 2 ORR AS AS IS AS IS A hs AS 7 ahs is Hs 8 os AS AS Ae As le hs IS oie 2 26 AS AG 2h ots 2h ie is AS Ae As As As 2h he “Be oho hs oie he as hoc 


// Global Variables // 


[FR AR AR A SE He He 28 he He Ae He 2 2s He SAS 2S AS Hs 26 AC He Bs As Hs HS Hs he AS Hs HF Ae AS He Mie Hs OBC OA AC A is AS AS As As Ae AAC As As As As 2s As As M6 MIS Hs Hs 246 2S 2K 2S Oe AS oho 7 


// NONE 


I a A i aa cet A A a 2 eR 2 2 oR 2 oi ie he 2 2 Oe 2 2 oe ok ok Ae OH OK OE 


// State Variables // 
[252 OR DA A A oR A A 2 OE a EE CA A ACA A AR OK 2 2 2 2 2 2A 2 ACC CE 2 2 2 HAC 2A 2 2 a 2 2 2 Ho oo oa a a 
/Aint d_id_list{200]: Destinations I might talk to 

/fint  subnet_list{200]: Their assoc. subnets 


/Ant © num_proc: Number of nodes in the system 


ara Nata I RR Na tee Parte oe ia che Aa aia hehe rhe erecta ota chet eta rc lehe cee refer rect hace re eke ee oe AC i KK 


// Temporary Variables // 
HE ea Ea See So Se lice ode oS ttt I Se Sol Sea lbs cco a Se Se a See Seta tes or Sa cS 
//Packet* pkptr: The pointer to the active packet. 

/Ant dest_index: The index used to access the arrays. 

//int dst_subnet: |The subnet assoc with the destination 

//int d_id: The destination. 


// end get_pkt 


Piatra ens eas eRe ake te tniets eis res ctu oF ote cts ofa che che ctncetocaieratataeengicty ego otc as ots cestctae xan delet Ae Ao mf ohc ote ots he oR oha halehaah oan Ca aie teeter ee 


14. B_TO_B STATE 


The following code is contained in the b_to_b state in the Fibre Channel node 


FSM. 


[FR PR AR Bi Ee A is 2 ie 2 ee OR ie 2 ie OK he 26 2 26 2h 2 2s 2K 2k 2c 2 2c 2 he 2S 2 24s 2S 2K 2 ke he 2S he 246 2c 2K 2 2 2 2c 2 2 2c 2 2c 2 OK 2 2K 2K 9 2 OK 2k 2K OK 2 OK / 


/[This state checks the buffer to buffer credit 

/Af you have no buffer to buffer credit the statisic 
//is logged and the buffer refilled 

/Af credit is avail, it is decremented and the pkt 


//fwd to b_plate 


if (buf_to_buf==0) { // is buf_to_buf =0? 
no_bb_count = no_bb_count++; //increment the number of times 


//that you've reset the bb credit. 


Op_stat_write(no_bb_count_sh, no_bb_count); //store the statistic 


buf_to_buf = max_buf; //reset the buffer 


[FPR a Ae ae AS AS 2 AS ote AG OIG AS he AG PE 5 OAS 0 AS hs AS 05 05 Ra hs 2 Ea St AS AG oR a he AS ti he A oh re Pie i 


// Global Variables // 
[PRCA AC ACC ACA CR IC CR CR COE IC ICA ICC RCO KFC GK CR CR CR CO 8 ROR COR COR 
/ NONE 


JAB EEA A Ee Oi 6 eG He ee Ae a AR RS oie AS ha Ss i a ea a in ie eee ie nie a re oe A ee 


// State Variables ii 


[FEE Be Be AA Ae i A A Be ots AS AS ic Ae Tle 2 Sh eI aera oe te ak Re AS AAS AS AS AS 2 Oo hehe RR A A eo oP re ee 


//Stathandle no_bb_count_sh: SH to the bb_count stat. 


//int buf_to_buf: Buf_to_buf credits. 
//int max_buf: Login value of buf_to_buf credits 
//int no_bb_count: Count of the number of times b_to_b is reset 


BEG OAS FAG 2 AR AG oR HG 2 AS 8 AE hoes AS 2S ES RE 2 SHAG 8 he I se sons 28 OS As ORS Gs ois le a =P ig rs Pie he chee a 


// Temporary Variables // 


[OR RR A ACR ICC ACCA RCC CO ICCC ACCA ECR I CCK CC A I COR CSCC EK ICC COCO OO AOI 7 
NONE 
/lend b_to_b 


[PRE PR AE AAG OR 2 OR A Ae ae Oe AE 2 Ae AS AE Ae eA Ae 2 AE Ae 2 Ae Ae 3 2 AS As Oe he Ae he AS 2 He ee 2 AS is 2 2 2 2 ie eo Oo eS he 


lS; B_PLATE STATE 


The following code is contained in the b_plate state in the Fibre Channel node 


FSM. 


[FRE PR AR AR Hee 2 he He HG A Oe 2 AC He 2 AC 2 Ae 2 ee 2 AS IE Dee AS GO 2 AS AC eA 2 2 A Ae 2 2 2 OE 2 2 2 ie he he 2 6 ie he 6 2k ae os AC Ef 7 
//b_plate sets standard fields in the header and sets the 


//size of the data payload and finally sets the class 


//of the message 


[BR ae he As eae le he ae ahs oe hs 2k 2c Ie As 2h ahs is 9s hs 2h As As ie AC 2h As He AS 2h An le hs ate hs a he As 2k lee ois Ae 2 og hs hs os is As of 


‘a The following is Frame boiler_plate | 


fT Eo Ra ie tan oto che Ae oe ole oh oe Se ok, es ate oo oie oe oe oh i eh de oe ich oh ee a ee ee 


124 


// These settings are the same no matter the class of the 


//message 


op_pk_nfd_set(pkptr, "SOF_0", OxFA); //K28.5 


op_pk_nfd_set(pkptr, "SOF_1", 0xB5); //D21.5 
op_pk_nfd_set(pkptr, "R_Bits", 2); 


op_pk_nfd_set(pkptr, "INFO", 2); 


op_pk_nfd_set(pkptr,"D_ID", d_id); 
op_pk_nfd_set(pkptr, "S_ID", s_id); 


/ This sets the src id field which is useful for sending 
//trdy’s back to the guy who sent you the frame. src id 


/fhas zero (0) length and does not affect throughput calc. 


op_pk_nfd_set (pkptr, “src id", s_id); 
op_pk_nfd_set(pkptr, "src subnet", subnet); 


//There is no rx_id yet so set to ffff 


op_pk_nfd_set(pkptr, "RX_ID", Oxffff); 


//Use attr_get to get the extended attribute Data Size 


// Data Size is the mean data size sent by the node. 


node_id = op_id_selfQ; 
op_ima_obj_attr_get(node_id, "Data Size", &data_size); 


/{Protocol is the FC-AE protocol so set Type to 0x4a per p. 188// 


op_pk_nfd_set(pkptr, "Type", 0x4a); 


//set the data field size, if you want MAX _DATA_SIZE for all Frames 
//Set the attribute Data Size to -99. 


Ps) 


if(data_size == -99) { 


op_pk_bulk_size_set(pkptr, MAX_DATA_SIZE); 
} else { 


op_pk_bulk_size_set(pkptr, data_sz_set(data_size)); 
} //fend if 


//set the creation time 


op_pk_nfd_set(pkptr, "Parameter", op_sim_time()); 


/{What class 1s the msg?? 


class_type = op_dist_outcome(class_dist); 


printf("The Class I get 1s %d\n",class_type); 


[FEAR Ae EE ee A a 2 OE ER SE AR SE SEC SE 0 2 AE SO 8 oh AS AC ha ha i 


// Global Variables ie 
[FEA AR PE OR 2 RE OR A AS 2 AS OS 2s OE a AR OAS a he ie oe AS AS AS IG he ohm eh Asatte eke chs obs choke che chm che ohn ete he oR Phe oho once ohm he Ax SIs oA oh “he char is 
//const int MAX_DATA_SIZE: In bits, the largest the data payload can be. 


[5 RPE A OE A OR A A OR OO HRCA OR RE RE A EE A E22 E22 EE 2 CE 2 E22 E22 3 3 2 2 OK / 


// State Variables // 
[PEPE RAE Ee Ae EAR AE AR AE OE AR Ee aR A A A Ae Ie ee ie Ae Ae oe Ae 2s 2 hs 2 AR 2 Ae oe Oe ois 2 AS oe A OE eo 2 EO 2 oS oe A oe 
//int s_1d: The source id of this node 

//int subnet: The subnet of this node 

[AERIS EEE Eas As eA os a RS oR a Ais OR Ap ohooh ols 2 ls Ae ois a oh oie ls oh Aa ole oe ots in ls ois is chs hs a in 2h oleh ats a ke oh 2s oA a 
// Temporary Variables He 
[ PRPR RAG AE Ae AR AG ee oe 2 oe 2 ae Ae Ae oe ie Ae Ac AS oe he he AE A Ae he AS hh fe Ae oA oR ee RE he AG RE Oe he he I Ae Ae OO EE PE Ee he i 
//Packet* pkptr: The pointer to the active packet. 

//Objid node_id: The Objid of this node. 

//int d_id: The destination. 

/Aint data_size: The mean size(in words) of the node. 

//int class_type: The class of the message 


[7 EPR HEH HH AR A HH OA OE ACHR OK OK A OK EO EE 2 EE 2 2 RCO 2 2 2 9 OE 2 a 2 a 2 a 2 oC Oe / 


// end b_plate 


[FR BR ee ie 2 he 2 2s 28 He ae 2A fe 2K he 2 2c he ae 2 2c ae 2K he ae 2c 2c 2K fe 2 fe a fe 2 fe 2 2A fe 2c fe 2 fe 2 fe 2 fe oe fe ae 2c Ac 2 fe 2 fe 2k fe 2 2 2k 2K 2c 2k 2 2c 2k | 
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16. CLI_XMT STATE 


The following code is contained in the CL1_xmt state in the Fibre Channel node 


FSM. 


Fst he ia ache Ree he a hs ar ect he i RR ce eh PO A Po eh te SR er ee rites Ac oie oi 7B os ee ote ri OE AS RC OR Ee ee 


//This state would be used to format class 1 messages 
//class 1 is not implemented yet (Aug 00) 


HE Stare Shoe StS i eae oc ai oti Su oS tenor as Sh Roh acti Sta 8 ae aaa Geshe | 


ie The following is SOFi1 p111 fc man */ 


nae enn hy che techn rte Pe As a he Asche bo Rete ee Ot ea eee iat rete get tere te eas ch ey 


op_pk_nfd_set(pkptr, “SOF_2", 0x57); //D23.2 
op_pk_nfd_set(pkptr, "SOF_3", 0x57); //D23.2 


printf("CLASS 1\n"); 


//CS_CTLis set in class 1// 
op_pk_nfd_set(pkptr,"CS_CTL", 0x00); 


HESS SE ote SE See Se eStart ost sa Se ok Se Se et fcr Se Sec St te Se So SE eS Esta aha Se Se Sage ce 29 ea ISR Sc tS eta ease Ske | 


I Global Variables // 
[PR ARR AE AR AE AGE AR AGE A A A GE OB A OE OR OE AR OE HH AG Oe A OR Os sO A 2 2s AG 2 2 2 216 26 His 2 6 2 oe 2 26 2 oe Ae OO AR oR AS OE OR / 
// NONE 


RS aoa tae St Sa ei ac asa eo ros a ca Se Su SSC ene Ses EEE EEG eS ee St Eos See St Secs Se Soe ar Sat arte ate co | 


I State Variables [| 


[PEPE AE AR AE AR AR A AG A 2 2 OE AG OE OK AG OR OE Ae AE HA HE AC AAG OR 2 2 AG 2 2 2 fe 2S 2 2k 2 2 2 2 28 2 2 2 ae 2k 2 2 Re GE EE / 


HI NONE 


[5 CR TT A AC ACR OK ACE ACR GR AC ER AC 2 HOKE HK / 


HI Temporary Variables /| 


[PR AR PR PRA AG Ae 2 HA 2 2 He 2 he 2 He CG 2 Ae OG 2 A 2 2 2 AG 2 CC AC 2 Ae 2 fe 2 AE 2 2 2 eC 2 fe ie 2 fe 2 2 2 ROK OE 2 Ae 2 OK OK OK 


//Packet* pkptr: The pointer to the active packet. 
/fend CL1_xmt 


[PR AE Pe 2 he 2 i 2 ie 2 2 ee 2 2 eS eS eC HO A OO OR 2g eH Oe IE 2 He 2 He EE He 2c fe 2 2 2 fe 2s 2s 2 2 2S 2 2 ie 2 2 OK OOK OK ee / 


bay 


We Gia XM STATE 


The following code is contained in the CL2_xmt state in the Fibre Channel node 


FSM. 


FR Foe FEE 20 A OE OE He 2 GE A SR 26 AE oe OE eR Te RE OR IE oe oie ote Ae Ee oR re ee 


//This state is used for formatting Class 2 messages 


[25 RRR RAH HH RR A A HE AC AC ACK A RR RR RR CE CC ACA A KK OK / 


ee The following is SOFi2 p1l11 fc man “i 


[252 RR RA A HH eR A AAC RCAC EC RR CC ROR A CCC HCE IE IKK / 


op_pk_nfd_set(pkptr, “SOF_2", 0x55); 
op_pk_nfd_set(pkptr, “SOF_3", 0x55); 


printi("CLASS 2\n"); 


//CS_CTL is not used in class 2// 
op_pk_nfd_set(pkptr,"CS_CTL", Oxff); 


J TOF ISIE AE AS 6 28 2 AS IE 2 Ae AS AG 2 he I AS 206 oh Ae IE IS AG 26 AG te AG ie Ske AS Ae A sec She te Sie ta Ie hs eke ote ote ete ee etn ate crt 


// Global Variables // 
[9 A He He He ee 2 2 He ee ee ee ee 2 ee 2 2 ie 2 2 2 2 2 ae 2 fe 2 fe 2 2K 2 fe 2 2 2 2 2 2 oe 2 2K akc 2 2 2 2k 2 kK 
// NONE 


[Pe 2 ee ee 2 2 RC 2 Re 2 A Ce 2 2 2 2 2 2 2 EE 2 2 2 eo a a 2 2 2k 2 2 2 a a 2 / 


// State Variables // 


PERS eae EE Ae EE Ae AE eae Ae EE Eee ee ee ee 


// NONE 


YS i a aS sg || 


/] Temporary Variables // 
JIE AE i ae ES he AE A Ae oe Aa ai ae oie OR in ie reais 2 oe oe oe ok iets as oh rhe inal ace ie oe teh aie re a he 


ji/ Packet” pkptr: The pointer to the active packet. 


/lfend CL2_xmt 


[EEE EEE EEE EE 0 2 Ee RR EE oe EO 2 As oe Ao oe te ke oie Oe ee te ec ee | 
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18. CL3_XMT STATE 


The following code is contained in the CL3_xmt state in the Fibre Channel node 


FSM. 


[5B RR SK HCA AK KA HE CC KK A A CA A ACK ACK KK 2K ok 2 OO / 


//This state is used for formatting Class 2 messages 


Bese ect ohe he otek ote tech a els che cy cae che rie hc ciel obe hohe obs Faint testa Rare ch ie hee e ee e a h 


ee The following is SOFi3 p1l11 fc man a 


[5 RR HA HK A KH RCA A EK A A 2 2K A A A HECK C2 2A FR C2 2 AC 2 2A 2 2 A 2 2K / 


op_pk_nfd_set(pkptr, "SOF_2", 0x56); //D22.2 
op_pk_nfd_set(pkptr, "SOF_3”", 0x56); //D22.2 


printf(’CLASS 3\n"); 


//CS_CTL is not used in class 3// 
op_pk_nfd_set(pkptr,"CS_CTL", Oxff); 


/ PO Sno Se i Re ce ae ce ia co, Se ac ee Ae eg ae ee ee eu te te eae i a eae Se roe Made es Ses a 


// Global Variables i 
[ESE Se oe Se SL Se ede Se Seaside teas Sock Seo 2 eae Se east Stade So Sett, Sead So) a ease ea tae So Se a ec Se SE a St Sent 
// NONE 


LO Se So SS ee SG a Se EU de SSCA Se OE ea ca da Sacco Sa So ee Se Se a at Seca tect cua. | 


// State Variables // 


[2875 A BR RR HK RR A 2K 2 AC AC 2 A 2 2A. 2A 2A 2 2K C2 2 2 IC 2 2 2A iE 2 2 2 2 ok 2 2 2K 2 2K / 


// NONE 


[PR PRR ARAB AGAR AR AB Ae AK He AE He Ae 2 Ae 2 6 2 2 2 2 Ae 2 ae 2 2 26 2 26 2 2k 2 ie 2 2 26 2 246 2 2 26 2k 9 6 2 a 2 2 6 2 2 26 2 Ae 26 ie 26 ie 26 ee / 


// Temporary Variables / 
[25 2K AR A A A A A 2 A ACK HH AC 2 EK 2 FAC C2 C2 FC 2 2 2 2 C2 2 2 2 2 2 G2 G2 6 2 2 2 2 2 / 


//Packet* pkptr: The pointer to the active packet. 


//end CL3_xmt 


[PR PR AR PR AR A 2 A 2 2 2 2 2 2 2 CE 2 2 2 2S 2S 2 2 2 2 2 fe 2 2 2G 2 2 2 2c 2 2c 2 2c 2 2 2 2 3K 2K he 2K he 2 2 2c 2c 2c 2 2 he he 2 2 
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1: E_TO_E STATE 


The following code 1s contained in the e_to_e state in the Fibre Channel node 


FSM. 


JPEG OR 26 OE As AS 2 i 9G ie 28 Se 2s OS 2B AS is As 2A Sets rie he a he ake angie Stn ote eh ieee ae ots oh ohn ets oferta ena teri eens da ote chai 


//e_to_e checks the end to end credit with the far side node 
//if there is no credit the statistic 1s logged and credit 


/Ais restored to its onginal login value. 


if (e_to_e(dest_index] ==0Q) { ~" fis end_to_end = 0? 
no_ee_count = no_ee_count++; 


op_stat_wmte(no_ee_count_sh, no_ee_count ); 
//store the statistic 


e_to_e[dest_index] = login_e_to_e[dest_index]; 
//reset the credit 


e_to_e[dest_index] = e_to_e[dest_index]--; //decrement the end_to_end credit 


JER AE A 2 ae oo oe 2 oo A eo te 2 ate re ote a le A oie Re oh oh ee ie Ae lea ee eo ee ee ee 


// Global Variables // 
[RC HRC ICC RCRA CICK AC ACRE KR RA CR AA 2K CK A OK 2 CR EC EC AE 
// NONE 


JER AE A EE OR A GA OE 2 2 eA oe 2 ee ie eR oe oie ee oe oe Ae oie ee ie eo ek 


/ State Variables // 


PR AB AB A DE A He 2 he is 2s 2 2 ae 2 he 2s fe Ae ie 2c 2c 2c 2s Ae fe fe 24s 2s 2c 246 2 2k 2k fe 2s 2 2 2g 2A 2k hs fe 2k 2c 2 2 24s 24s 2K 2k 2 2c 2 2s Ac oe 2 Ae 2 2 2 he 26 2 2 EO 


//Stathandle no_ee_count_sh: SH to the ee_count stat. 


//int no_ee_count: Count of the number of times e_to_e 1s reset 
//int e_to_e[200]: End_to_end credit of the connected nodes. 
//int login_e_to_e: Login end_to_end credit 


130 


[7 RRA RR CIC A I AR ICICI ICI I ICRC CI ACR ICICI RCC IC IC AE ICSE SI ORK CCK J 


/ Temporary Variables // 


a Ra Te a i a ale ata aaa ie oe A IR oe oe eR RE AR oe He oR oo A OR EE | 


//int dest_index: The index used to access the arrays. 


//end e_to_e 


[28 RR RRA EC ES HH A CCE AC CEE 2 ACC EE A 2 2 2 EOI EEE IEE 2 EEE EA ACO ACC OK / 


20, ROUTE STATE 


The following code 1s contained in the route state in the Fibre Channel node 


FSM. 


YFP Ae ote 2 Ae oR Ae Ae A EE A ta oR a Ae oR AG AS a ote 2 a a Aa ri te a a ee ee ee) 


//route is responsible for adding the route to the packet 
//First check to see if the rte_arry[] is null 
//This will tell you if the route has been built already, otherwise use the SV 


//rte_arry[dest_index] to make the tmp_rte which 1s added to the packet 


if (rte_arry[dest_index] ==NULL) { 

rset_ptr = op_rte_routeset_create (TOPO_PTR, subnet, s_id, 
dst_subnet, d_id, MAX HOPS); 

tmp_rte = op_rte_routeset_mincost (rset_ptr); 
op_rte_route_print (tmp_rte); 
rte_arry[dest_index] = op_rte_route_copy (tmp_trte); 
op_rte_route_print (rte_arry[dest_index]); 
//destroy the now useless route set pointer 


op_rte_routeset_destroy(rset_ptr); 
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Op_rte_route_print (rte_arry|dest_index]); 
tmp_rte = op_rte_route_copy (rte_arry[dest_index]); 


op_rte_pk_route_insert(pkptr, tmp_rte); //add the route to the packet 


[FEE ER Fee AG i6 He BS AS PAG ee AI che i hc eat ere rasta the Aa cee he aec Ae 28 ik Cis ie oe eee he eR ae 


// Global Vanables // 
[75 AR A HH A EH ACH EC A AAC AE 2 A AE AE 2K A A EC AC CE 2 oA CE 2 2 EE 2 2 2 2 2 oo CK / 


/[Topology* TOPO_PTR: The only Topology Pointer 
//const int MAX_HOPS: Max number of Hops a path can have. 


Yi ee a ie ae aa se to St Sk tat es Se Se Se at at Sse Sk eset eset ao 


// State Variables // 
|i ea Se ee a a se a So ee Ct ot St ot Sock Se So SE Src 
//Route* rte_arry[200]: Holds routes from this node to all others. 

//int s_id: The source id of this node 

//int subnet: The subnet of this node 


[ERR A RR AG He He oe re i oe oR oe Ae Ac A os ois He oe Ae hc is ahs ohn ee ie 2k Ae Ae ae Eh ce ee 


// Temporary Variables // 
[FREE Ae A Ae ie Ee Ae He Ae ee 2 Ae ie Ao AS He se Ae Re Ae he 9 Ae hs he As Ee he ae he coe 8 ee he oie he ote ha ee ie oe 2 ee 


//Route_Set* rset_ptr: Pointer to the routeset. 


//Route* tmp_rte: A copy of the route to be used. 

//int dst_subnet: The subnet assoc with the destination 
//int d_id: The destination. 

//int dest_index: The index used to access the arrays. 


//end route 


JPR EEE Ae He oie oi Ae ee ie oe eee oe eo ee oe ie 2 eR ee ook ee ee hae ee ee 


Zi), XMT_PKT STATE 


The following code is contained in the xmt_pkt state in the Fibre Channel node 


FSM. 


[FER A A OK He 2 Oe 2 2 ae i Hee he 2 fe 2c eG fe 2 eof 2 fe 2c fe 2c 2 2 He fe fe 26 2 2 ak fe 2 ofc 2 fe fe 26 2 2k fe 2 2K ie 2c 2 2 2 2 2k 2 2 2 2k ok 2 


//xmt_pkt is responsible for sending the packet 


//out of the node with or without delay 
132 


buf_to_buf--; //decrement the buffer to buffer 
if (proc_delay '=0) { //if the process delay is not zero then 


//send it out the transmitter with the proscribed delay 
// the following code allows a PDF (exponential is 
// used but any PDF could be put here) to be applied 
// to the delay 


// op_pk_send_delayed (pkptr, XMT_OUT_STRM, 
// op_dist_exponential (proc_delay)); 


// For testing and verification a constant delay was used 
op_pk_send_delayed (pkptr, XMT_OUT_STRM, proc_delay); 
\ else { 
//send it with no delay 
op_pk_send (pkptr, XMT_OUT_STRM); 


TE SE Se eS SE Se SS SS SS Se a Se Se SSE ga SSC Se TSR at Sica Sits Seca Sime I) 


// Global Variables // 


Se cE Se SS a ES a SS noe cone Seats Seas ct ncn eas ost Catt Soe Sys toe as eats in can || 


// NONE 


TEs BE sie Se Ie See SG aaa eo ee Be SEE oe Se Pee See SC Ce See ce Pee SG ae ce eee co ec Se Se oa 


[/ State Variables // 


[RR ARR RA I TK KR RK KR OK HE OK HK KA RA OK A A A OK A KO AK / 


//double proc_delay: The processing delay associated with this node 
//int buf_to_buf: | Buf_to_buf credits. 
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ff ASR Re 2H HS ee 2 he 2A eA I i aha cpa acre ate ee 2E 2 Re AS ES A Ie he I ie hee iia ete eee a 


// Temporary Variables // 
FT aa ea a ea ac eet eee ae Se ia kos ala ste diy scosi ead tase stou. tac acasiugtacas 


Higackel” pkptr: The pointer to the active packet. 


/fend xmt_pkt 


Ye aaa ee aaa a a a a ee et Se eS ee Ae ia eae euetoiee eased Seis 


22 RCV STATE 


The following code 1s contained in the rcv state in the Fibre Channel node FSM. 


[A PE A HR HK He ee KE 2 ke 2k ie 2k Ge fe 2 fee fee fe 2 fe 2 2 ke 2k ee ae 2 fe 2 2k fee 28 fee 2 ok 2k ee 28 2 ke 2 2k fe oe 2k ok eo 2k ok / 
//rcv state is used when a packet 1s received 


//on the node receiver (as opposed to the pkt generator) 


//state figures out if pkt is a frame or primative 


numFields = -1; //init to ensure proper xsitions 
c3Rrdy = -1; 


/* get the packet oh 
pkptr = op_pk_get (RCV_IN_STRM); 


op_pk_format_info_get(pkptr, 
OPC_PK_PROPERTY_NUMBER_FIELDS, &numFields); 


if (numFields <3 && numFields > 0) { //if less than 3 it is an rrdy 
coknay = I; 
} else = { 


c3Rrdy =0; /if >3 itis a frame 
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[RR RRR AR A HT HE CR HE EEE ARC EE IEE 2 HCA AE 2 2K 2 2 22 2 EE a CC I HH HE 


I Global Variables // 

[BERR SEAR est Fe SESE SESS SE See eS Se cascode Sic ee eae ea oc oe ac aod ae aaa ee EE kk Ek 
// NONE 

[9525 2 2 2 EEE EE EEE ee A AACA FE 22 EE 2 HE EE EE 2 EEE AE A ACK 2 2 EE EO Oo a a CO / 
// State Variables // 

[95 ATE RC EE EE EE RAC EE Ae 2 EE 2 AC EE 2 2K 2 2K 2K 2 HE 2 2 2 2A A 2K 2 ea a a 2 2 2 2 / 
// NONE 

[BE RR HE ACE eC CHE 2 2 2 2A 2 2K Ce 2 2 Oe 2 2A ke 2 ACA HE 2 24 2 ke 2 2 2 2 2 2 EO / 
// Temporary Variables // 

a ss i cat Se a Se ea ae cc Ge eg Coe cae Seo aa eu case coun ao od tke es Sea se Esa tea Si esc Se de 
/[Packet* pkptr: The pointer to the active packet. 

/fint numFields: The number of fields in the frame. 

/fint c3Rrdy: Indicates the reception of an R_Rdy 

//end rcv 


Re aa ac ee a a ca a a DL i at ate ice a ce a Ne ae ae ee 2S ae se use Seo) 


Is R_RDY_RCV STATE 


The following code is contained in the r_rdy_rcv state in the Fibre Channel node 
FSM. 


[ER RH ER A ER RE RE RE A A OE A AE RE EE EE 2 EE OC DE a 2 OE CHE / 


//r_rdy_rec is used when a primative is received 
/if an r_rdy was received the b_to_b credit is icremented 


//provided that the credit isn't at max already 


if (buf_to_buf < max_buf) { //if the buffer isn’t maxed out 


buf_to_buf++; //add one 
} elscun 4 
buf_to_buf = max_buf; // otherwise leave it a max, because credit cant 


//be greater than the max. 
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op_pk_destroy(pkptr); 


[EER EE EE A Ae 2 OG 2 HE EG 2 A Ee Ae OE a a a Ae oe a A a eee 


/| Global Vanables /| 
[EEE RE A A Ee EO A 8 A AG hs a A he eI heh a Ae oR Ne 2 ao ie 2h OR oe a oh eee i ee 
// NONE 

fie os a As oS oR 6 a As 6 Re 6 AS AR OR is IS ates atrazine teas eer os ore toca acetate cet ok sols As 9 oie OF 20 18 ols oh ohn 2 he ho os Pak oho ti 
// State Variables // 
[REAR EAR EA A AG sR As BO AR as A ah Res AG As A lalla Baie ob le ots che Pe ths che os ols Pe ols vin ale ats is ols chs is lsc: oie cig iy a 
//int buf_to_buf: Buf_to_buf credits. 

//int max_buf: Login value of buf_to_buf credits 


[FE 78 RR RA EE Ae AE EE 2 hs he Oe Ie ee i ee ek ee 


// Temporary Variables // 
[FEE A Ae 2 Oe EER EE AE eae Eola eke oh Ee ore ek Eee ee ee ek ee 


/[Packet* pkptr: The pointer to the active packet. 


//end r_rdy_rec 


PR PR AE AE 2 2 ee AG 2B Ee 2 OB Ae He 2B Ae 2 Ace Ae ae 2 Ae 2 oA he he 2a ie oe 2 fe Ae 2 he 2 fe ae oh he as Ae a he ote Ane ote ls oie ois oh 2 oh oie rea 


24. SEND_RRDY STATE 


The following code is contained in the send_rrdy state in the Fibre Channel node 


FSM. 


PEPE ARE EE AEE Ae a a a Ee 2 EE ee or ee hee eh ie ie late i eo ek ee ee 


//snd rrdy builds an rrdy and sends it back to the previous node 


//that sent the frame. 


pnintf("I got a frame!\n"); 


rdyptr = op_pk_create_fmt ("R_RDY"); 
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//get the src node from the frame and put it into src_nde 


op_pk_nfd_get (pkptr, "src id", &src_nde); 


//put your s_id in the "src id" field 
op_pk_nfd_set (rdyptr, "src_nde", s_id); 


//calculate the delay 
//check for delay 
fiierdyedelay 0) =| 


//send it out the transmitter with the proscribed delay 
/ op_pk_send_delayed (rdyptr, XMT_OUT_STRM, 


// op_dist_exponential (r_rdy_delay)); //delay with some 
distribution 


op_pk_send_delayed (rdyptr, XMT_OUT_STRM, r_rdy_delay); //constant delay 
} else { 
//send it with no delay 

op_pk_send (rdyptr, XMT_OUT_STRM); 


// get it’s R_Bits and assign it to fc_type 


//so that we can transition to the next state 


op_pk_nfd_get(pkptr,"R_Bits", &fc_type); 
printf("Done with snd_rrdy\n"); 


[BARRERA RAK KKK HAA KK HRA A HR OG a 2 Oe 2 Ae Oe Ae OG A OO He 2 Oe OK Ae Ee he 2 ef 


// Global Variables // 
[PRP AEA A Ae 2 EE 2A A OR AE AS Oe 2 ire oe He 2 Ms rs 2s so is oh iat ha ia she 08 8 2 i 2 Ae oh ote Ae ae oho eo 
// NONE 
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[RE re ER oe AE he oF ee AE IE i he A aE ie he ah 


// State Variables // 


[7 RR ee re rie oe ee i eR A Ae A ie ore 2 eI eA ee i oS ie a i 


//double proc_delay: The processing delay associated with this node 


//int s_id: The source id of this node 
J ear oe oe Ae A a A EOF eA AGS AE ho A a ho a oh oe oh ake ag cache eye ohe hy ae cred te che che hy onc hercty ase ctera, a ote Re ct 


// Temporary Variables // 
PEAR ds He le oie Fe 2 2s 26 2 He 6 2 ie a AR He IG os Ae ae Ae os ok ats as ahs oo Os ois sats ache ois os Ak oh A okey os ha he ce hs recto teh 
/fPacket* pkptr: The pointer to the active packet. 

/fPacket* rdyptr: The pointer to the created R_RDY packet. 

//int src_nde: Put into the src_nde field for ease of retum 

//int fc_type: Is it FT-1 or FT-0? 


//end snd_rrdy 


[eR ee a AG A a Ae OF a 2 ie Ae ae ie 2 2 2 Ae AG Ae Ae 2 2 eR he Ae As oA oe he AS ho Ao ie Rol SR oh oh ok ee 


2o7 FC_USTATE 


The following code is contained in the FC_0 state in the Fibre Channel node 


FSM. 


[FRR FERRE RAG eA 2 Ae oe ARE 2G 2 2 OE OE AE 2G AG 2 Os 2 2 2 AS 2 28 A oe AG oe 2 A ae ie oi Ae oe ic of ak oe 2c 2c hn ak oie 2b 2h ie hoa 2 As 


//The FC_O state is entered when a frame type zero 
//message arrives at the node (FT-O are link control frames) 
//Actions are taken based on the type of LCF that is 


//received. 


// find out where it came from and increment that 


//value of e_to_e[i] 


op_rte_pk_src_node(pkptr, &src_subnet, &src_id); 
for (k = 0; k < 200; k++) { 
if (d_id_list{k] == src_id) { 
printf("E to E for %d is %d\n", src_id, e_to_e[k]); 
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emtOwe | tole store(s 4- 
break; 


printf("E to E for %d is now %d\n", src_id, e_to_e[k]); 


op_pk_nfd_get(pkptr,"Parameter", &create_time); 
ete_delay = op_sim_time () - create_time; // calculate the end to end delay 


op_stat_write (ete_gsh, ete_delay); // write the statistic 


//find out if itis an F_BSY or not 
op_pk_nfd_get(pkptr, "INFO", &f_bsy); 


ies e—— XD) { //Af it’s an f_bsy to a data frame 
F_BSY_DAT = F_BSY_DAT++4; //increment the stat 
//printf("F_BS Y_DAT IS %d \n", f_bsy_dat); 
op_stat_write (f_bsy_dat_gsh, F_BSY_DAT); //wnite the stat 


} else if (f{_bsy == 0x6) { //t’s an f_bsy to an LCF 
F_BSY_LCF = F_BS Y_LCF++; // increment the stat 
//printfC"F_BS Y_LCF IS %d\n", f_bsy_lIcf); 
op_stat_write (f_bsy_lcf_gsh, F_BSY_LCF); //wnite the stat 


} else if (f{_bsy ==Ox2) { // the frame got there and back so calc e-to-e delay 
op_pk_nfd_get(pkptr,"Parameter", &create_time); 
ete_delay = op_sim_time () - create_time; 
printf("Create Time = %f , Sim Time = %f, Delay = %f \n", 
create_time, op_sim_time(), ete_delay); 
op_stat_write (ete_gsh, ete_delay); 


} 
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op_pk_destroy(pkptr):; 


PB NR Oe ee aie oe AO A A I A cr i a ot ae tao ae ee ee 


// Global Variables // 
AE AR A Ae Ae OE AR AE EE AG I ae 2 OR OR IE GR 2 Ae OR ie ae ee 8 ee 
/Ant F_BSY_DAT: Holds data for F_BSY’s to data frames. 
/Aint F_BSY_LCF: Holds data for F_BSY’s to LCF frames. 


J RG A As OE Hs 2 AS 216 He a Ess Hs ASH iS 0 ahs a hp 2 2S AS hs ote 2 ls ate cha eo Ps acts 2 chet is ete cha oh He ots 2 rg eleinky cleo hs rie chs ote ciara 


// State Variables // 


[RFR 26 a a8 As 2 AG 2 AG 2 is 28 2 RG AR ER RAS 2 aS oS 5 28 AS aS vhs mayo fe hes he ee eb hs 2 chy faethe ols ie ety a ots oh rhe ras cts he els ty ctv alco 


//Stathandle — ete_gsh: SH to the ete global stat. 
//Stathandle f_bsy_dat_gsh: SH to the f_bsy to data stat. 


//Stathandle f_bsy_Icf_gsh: SH to the f_bsy to LCF stat. 
/int  d_id_list{200}: Destinations I might talk to 


/int  e_to_e[200]: 


End_to_end credit of the connected nodes. 
[FPR AR ER AE 2 A EE AE 2 ae he 2 eA Oe I i oR tan ot ore re eee 


/ Temporary Variables // 
[Pa oie We ee ee AS He Ie 2h 2A 2 hee 2c ote ale AG He als Be ake en of 2h aie he ahah oe oie Ae hs che 2 Ao eae ee 
//Packet* pkptr: The pointer to the active packet. 

//Aint src_1id: The source node. 

//Aint src_ subnet: The src_id’s assoc subnet. 

//int f_bsy: Used to check if the packet was an F_BSY 
//double create_time: The time the packet was created. 

//double ete_delay: The time it took to get here. 

//fend FC_O 


[RPE AR AR AE eR ee oR Ae OO AE 2 2c 2 IR ee ie i eke oR 2 OF Fe 9 hohe a obs fe tate le fy of ie BaPAG ok ole oh eta ee ote ote ois Pe ae ate chee a ea 


20, Peo lA te 


The following code is contained in the FC_1 state in the Fibre Channel node 


FSM. 
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//FC_1 state is entered when a data frame (FT-Q) is 
//received by the node 

//This state returns acks if nec, otherwise it calculates 
//end to end delay for class 3 msgs. 

// init the transition variable 


class_3_rec = -l; 


op_pk_nfd_get(pkptr,"SOF_3",&classT ype); 
printf("%d\n" ,classT ype); 


if (classT ype == 0x56) { /Af it’s class 3 no ack is rqd 
//ete_delay = op_sim_time () - op_pk_creation_time_get(pkptr); 
//op_stat_write (ete_gsh, ete_delay); 
printf("here\n"); 
op_pk_nfd_get(pkptr,"Parameter", &create_time); 
ete_delay3 = op_sim_time () - create_time; 
op_stat_wnite (ete_gsh_cl3, ete_delay3); 
op_pk_destroy(pkptr); 
class_ 3 rec=1; 

} else if (classType == 0x57 || classType == 0x55) { 
//else if it’s class 1 or 2 return an ack 
//where it came from and where the fc-0 has to go back to 


printf("I'm returning the fc-O now\n"); 


op_rte_pk_src_node(pkptr, &dst_subnet, &d_id); 
for (k=0;k<200;k++) = { 
if (d_id_list{k] == d_id) { 
dest_index = k; 


break; 


14] 


//ouild the ACK for sending 

op_pk_nfd_set(pkptr, “"D_ID", d_id); 

op_pk_nfd_set(pkptr, "S_ID", s_id); 

op_pk_nfd_set(pkptr, "R_Bits", OxC); //make it an ack 
op_pk_nfd_set(pkptr, “RX_ID", 0x00); //give it an exchange ID 
op_pk_bulk_size_set(pkptr,0); //make the data size zero 
op_pk_nfd_set(pkptr, “src id", s_id); //for sending rrdys 


class 3_rec =0Q; 


‘ //end else 


[FE PEPE AE A A IE ae a oa a a ea I i a AS Ae enh Ae he Ao te ech cre ak ae ei 


// Global Vanables If 
PR AE AR AR RAG oR a oi AR As MS AG hs oR ie sR oh We ee hs a os OE ols 9 ols oie ote ahs ols he Bnet 9s As chs ets Plea rhe obs 2 eis ole oh on As ois chs ots ahs os A oto ee 
// NONE 


JS A AS AES PIS 0 OS HS OAS OAS 2S Ss ohm oe SEIS See HS SAAS hs PRS SAS Phe HS. chs ok ota As oh wtx AS che che cone eis cha chs chet ie iach ee re ae hohe ote cha oho tect a 


// State Variables /| 


[PEAR AFR EAE ARR AR AE OE AG A a AG A PE oi oA oho eels oe oe Pharaohs the hacia chs oh ohn vhs Ts os obs ety es As charts ete ele chy cas ahs An ivheirhs oh hs ce 


//Stathandle ete_gsh_cl3: SH to the ete_cl3 global stat. 


/Ant  d_id_list{[200}]: Destinations I might talk to 

/Ant = s_id: The source id of this node 

[AA A A HE AH HE A AE A HK EK AC A A GE AC A 2 2A 2 A 2 A 2 A 2 2K 2 A 2 2K 2 2 OK OK / 
// Temporary Variables I 
of RAGE AS I AR A AE Pi ae MES Hes hehe ote ts Sh AS oA es S/he obs 8 AE he 246 ORAS AS rh oS Poke te GS AAS ha AS AS AG he OA Alo haces ree arn crete 
//Packet* pkptr: The pointer to the active packet. 

//double Create_time: The time the packet was created. 

//double ete_delay3: The time it took to get here. 

//int dest_index: The index used to access the arrays. 

//int dst_subnet: The subnet assoc with the destination 

//int d_id: The destination. 

/fint class_3 rec: Wasaclass 3 received or not. 

//int classT ype: Holds the class of the frame. 

//fend fc-1 


[A PRP HH A HH A a RE A A RC ACE A AC 2 AC A HE 2 2K 22K 2 oC A 2 2 a 2 HAC EK / 
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Zi CHECK CREDMESTATE 


The following code is contained in the check credit state in the Fibre Channel 


node FSM. 


[PRE I  R HR A R  R A R  EE EEE CE R A C ACH  S ACH A A HE K / 


//check credit makes sure that there is adequate b_to_b 
//credit avail to send the ACK back 


//if not, the b2b stat is incremented and b2b credit restored 


if (buf_to_buf==0) { // is buf_to_buf =0? 
no_bb_count = no_bb_count++; //increment the number of times 
//that you've reset the bb credit. 
op_stat_write(no_bb_count_sh, no_bb_count); //store the statistic 


buf_to_buf = max_buf; //reset the buffer 


} 


fore oh Ae che rhe ohs he Ae 2B ete he oF oh AS oe “Re ok 26 Ae oR 2 Ae 2 fe OR a Ae Ae oR 2 fe oe oR fo Re eS ooh 2 AE Re he Aa As oh he iG alo ote ck tee ee Scherr oh Ahh / 


[| Global Variables // 
PEE eI asec Sea Se Se ca aa ee oa ute aR Sick cease eas a a dade See Seca sec 2 a See ce Sa SES Se Se Schl 
// NONE 


Pa ee ae eae Se Se cesT Seat ear tas cos aes code nC Gs cco en Stace de Soasaeu ea Raagde Sete Tee Je a dataset st | 


[| State Variables // 


UE Fe Se eS 02 oct eae asad Sou ad gs casas Cu sty ea Seth Sh See a Seu S Ege aees eas 2t eae cE Seode ate ee Se ase Se | 


//Stathandle no_bb_count_sh: SH to the bb_count stat. 


//int no_bb_count: Count of the number of times b_to_b is reset 
//int buf_to_buf: Buf_to_buf credits. 
/fint max_buf: Login value of buf_to_buf credits 


[959 RR A ER RCAC A RCAC ACE CE 2 AC EC CE 2 CE 2 EE 2 2 2 2 2 2 2 oO 2 / 


// Temporary Variables | 


[FR AR PR E AG  e 2 e A  e 2 e e 2 2 AE O e  2 2he 2 he 26 2 EE he 2 2 Ae 2 ae 2 Oe eo 2 2 2h 2K fe ee 2 2 2 2 2c 2 Ko HK oe 2 He CK / 


// NONE 


/fend check credit 


[P75 OR 2 2 ee he ie 2 ik he he 2 2k a 2 ie 2c 2 is aK Oe 2 2 fe He a 2 ee 2 ee 2G 2 He He OE 2 Ke 2 2 fe 2c 2K 2 2 a 2 2c 2 2 2 2K 2 OE 2 EE 
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APPENDIX C. OPNET SWITCH FINITE STATE MACHINE CODE 
l. LIST OF STATE VARIABLES 


Table C-1. List of State Variables Used in the Switch Finite State Machine 


xmt_id[32] int con_out 
| tink iaf32) | Ovid __|_ rd, sndrrdy, xmt, con_out, dec_buf 


util{32] float init, set_link_cost 
































total_ bits double sndrrdy, xmt, init 


rte_arr Route * add_route 





rset_ptr Route_Set * add_route 


r_rdy_dela | objid?, sndrrd 








proc_delay double | objid?, xmt 





max_credit int r_rdy, init 
ded_cl_3_gsh Stathandle 


2. LIST OF TEMPORARY VARIABLES 





init, dec_buf 


Table C-2. List of Temporary Variables Used in the Switch Finite State Machine 


subnet_id subnet? 


int te) 


pkptr hackel & type, r_rdy, frame, sndrrdy, 










xmt, f_bsy, add_route 








type 
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sede | ime pie 













next_subnet frame, add_route 
_nextnoce | _—_im* _|__frame, xmt, dec_buf, add_route 


temp_sid int 
int 
int 



















dst_subnet add_route 


im | add route 
pig ime ad route 
a ee ee 
pie inde 


5, HEADER BLOCK 


The following code 1s contained in the header block of the node’s process model. 
//Define all transitions as macros 
#define PR_ARRVLI ( op_intrpt_type () == OPC_INTRPT_STRM ) 


#define RRDY (numFields < 3) 
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#define FRAME (numFields > 3) 

#define UPDATE_COST (op_intrpt_typeQ) == OPC_INTRPT_SELF) 
#define CREDIT_AVAIL (credit == 1) 

#define NO_CREDIT (credit == 0) 

Hdetine Nimo Kiem eon ciasss —— 1) 


[25 PR A A EE EE AH HR HO CA KR HEC ACA A RA 2 A 2 EA eC ACA AC 2 2 EC CE EE 2 Go 2 oO CK / 


// Global Variables // 
Bae he he leicester in ay ete oe he eels he rl cheats hs i hse bs es oh a cin cla ts lag te che ck tale he hohe oe ate zich rte hee hs ois os ook oh ois hs 
double START_TIME = -1; //start time of the simulation 
const int BAND_WIDTH = 1062500000; //BW of Fibre channel 
extern Topology 71 OROL PA //The one and only Topology pointer. 
extern const int MAX_HOPS; //Max number of hops a packet can take. 
int DED Gis3: //Holds the number of Class-3 packets 
//destroyed by switches. 
const double DEL _ COST =.1; /{How often you update the cost of the links. 


ph Beaches ake he Fe She Aa ole An ke ahs eas ats 2 AS ofS of 215 AS ahs he fe AC 2h ie che Fs As ots oes fe ote 086 oF a oie fs ake 2 2Ae 2p abe of 2 cAaiohs ote okt fee laa ei a os ie i AG haces ors 


// Function Prototypes // 
ade ats ale Aa sctnscke ote ake Aa ols cke he ets AG ee eh eke A Ne ee 2X Ae se oh fe ofS ahs ah ech ek cha oka is oboe che che cle cha chy hci ote che haha enraged cre eer 


//Function: get_outStream 
//Purpose: Returms the outstream to reach the node of interest 


int get_outStream(const int [], const Objid [], int nde); 


/fend Header Block 


pf 7 2 oie A 2 2 Ae AR 8 Ae AS ee AS 2S 8 8 28s IR IS 8 PIS PAS os an OS 2A AS 2S he ha Ie ceicte ke oe Ae oP a 2 Ae ho heat Ae Ale Pa Ae ote crate ieee eet 


4. FUNCTION BLOCK 


The following code is contained in the function block. 


[FR 76 2 8 Ae 2 2 ee 2 ee he 2 oR ee 2 2 he 2 2 OF 2 2 2 Ae 2 2 2 2 2 2 2 2 2 a 2 Oe 2 2 2k 2 2 2A oe 2 2 2 2k 26 2K 2 OB Ee 2G 2 2 2K EE EK / 


//Function: get_outStream 


/{Purpose: Returns the outstream to reach the node of interest 


[PR PRP He ie he Ae ee oe OO 2 28 ie 2k fe 2 ie 2k 2 OR 2 2k 2 2 2 2 2 a 2 2 2 2 a 2 2 2 2 2 2 2 2k 2 i 2 2 2 2k a 2 A OK OK OK 8 2 OK / 
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int get_outStream(const int nde_ary[], const Objid Ink_id [], int src_nde) { 
inti =0; //counter 
Int out; 
double cost; 


double low_cost = 100; 


for(i = 0; 1 <= 31; 1++) { 
if (nde_ary[i] == src_nde) { 
op_ima_obj_attr_get(Ink_idf[i], "cost", &cost); 
if (cost < low_cost){ 
low_cost = cost; 


Out =1- 


} 
| //end for 


return out; 


} //end get_outStream 


[PRP NAR ee oi Os 2 Ee ae 2 ic a a oie Ac ois As ae oie is 2 ee Ae As eho he Ae ois oe ose oie le oes of ae As a aac oie As ote AA eee 


3) INIT STATE 
The following code is contained in the init state in the Fibre Channel switch FSM. 


[RRR RAE He BRR IH HH AE A Ae He 2 ee he 2 Ke 2 He 2 he 2 I Ae ae 2 Ae ae ee 2 2 he 2 he 2 Ae fe ae He he 2s 2 2 2c 22 2s ke Ae ai ake ais 7 


//Init state is used to do initial admin tasks in the 


//switch. This would be like a login in Fibre Channel 


//establish the buffer to buffer credit with each port. 
op_ima_obj_attr_get(op_id_self Q, "Max Credit", &max_credit); 
printf("Max Credit = %d \n", max_credit); 

for G = 0;1 <= 31; 144) { 


buf_to_buff[i] = max_credit; 
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//set first link cost setting interrupt 


op_intrpt_schedule_self(op_sim_time(),0); 


if (START_TIME < op_sim_time(Q)) { 
START_TIME = op_sim_time(); 


//init the Global Statistic var’s only one time! 


DED_CL_3 = 0; 


//Register the statistic 
ded_cl_3_gsh = op_stat_reg ("Dead Class 3", 
OPC_STAT_INDEX_NONE, OPC_STAT_GLOBAL ); 


for (i = 0; 1< 32; 14+) { /Anit utiliztion and total bits 
util[i] = 0.0; 
 total_bits[i] = 0.0; 
j 


[RM RRA A He eo Ae He ee oe 2 Ae EE oe 2 OE A OR A oR OR A HK A RO A RE OK HR OBE oR He 2 A OR A A oR Ee OR EE AE Hf 


// Global Variables fe) 
fo chs ohn oh oh obs Ae de 2 2 be oe oR he oe Aa oe ote alot EAR Ere Fe ere ee ee ee ee ee 
//double START_TIME: The time the simulation started. 

//int DED_CL_3: Stat for number of class 3 frames destroyed by the switch. 


[PRR A KH KKK EH EK A RA KH HK HH AH HH HH HH Hh A A Hh HE HE A He A AA He AE A 7 Ee EE Ae EE f 


[| State Variables // 
[78 RE A A A A 2 a A 2k 2 Oe 2 Ee 2 2 oe 2 2 26 2 2K 2K 2 9 oe 2 a 2 KK oe 2 a 2 2 oe 2 a 2 2 oe 2 i 2 2 2 2 2 2 ae ae 2 2 0 2K 2 2 2K HK / 
//Stathandle ded_cl_3_gsh: SH which holds the value of DED_CL_3 


//double —_total_bits[32]: Total number of bits to pass through the assoc link. 
//float util[32]: Utilization of the link. 


//Aint max_credit: The login buf_to_buf credit. 
//int buf_to_buf[32]: Holds buf_to_buf credit of the attached nodes. 
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[EP 2B EE AR he he iss Ae he Ae 2 Ae he Ae 6 216 1G OE IG ES as hss AS Ps °F Re eka cha IR oe tt isa chy het eg heehee eat te teeta tetera ay 


// Temporary Variables // 
Gf I Ae ee ee ae A Aa oS 2s AS oR 2 AE 2 I IR he oh ole Zi ine ose os icles intact cre As oor Ae a RIG ore ts a is oh aerate ee 
// NONE 

//end init 


J [FRE EEE EE EO AR AR GR A 2G 2 IES Ss EO OS AR Shs AS ahs AS he is oa es ie abe Ws AS OG sos ts hse cts IS ee che oe oie ie eich ha 


6. OBJID? STATE 


The following code is contained in the objid? state in the Fibre Channel switch 


FSM. 


J [FER AE EE AE Me SO Ae Foie He 2 AS He ae IS ie ie HG AG TG Hs aie Ae ie ak 26 Be AS As 2 ie hc 2he hs Is 8 2s 2h haa ree Ae 2s 2 ho he AC Ci ois cor oh ie 


//objid? state determines the objid of this switch 


//it also gets the delay parameters 


temp_node_id = op_topo_parent(op_id_self()); 

s_id = temp_node_id; 

printf("\nI am S_ID # %3d \n", s_id); 

op_ima_obj_attr_get (temp_node_id, "model" , &node_type); 
Op_ima_obj_attr_get (temp_node_id, "name" , &node_name); 


printf("my name 1s %s \n", node_name); 


//get the process and R_LRDY delays 
op_ima_obj_attr_get (op_id_self (), “Processing Delay" , &proc_delay); 


op_ima_obj_attr_get (op_id_self Q, "R_RDY Delay" , &r_rdy_delay); 


printf("Proc delay = %f ", proc_delay); 
printf("R_rdy delay = %f\n", r_rdy_delay); 
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[252 FR A A A HH HA HA A A AA A Hee 2 A 2 EC A oR ACA Kf A A AE 2 2 HE kK ok a KCK / 


// Global Variables // 
// NONE 

// State Variables // 
//Aint s_id: The source id of this node 

//double proc_delay: The processing delay associated with this node 

//double r_rdy_delay: The delay assoc with sending an r_rdy 

// Temporary Variables // 
//Objid* temp_node_id:The Objid of this node. 

//char* node_type[10]: The type of node (i.e. proc, disp, ant) 

//char* node_name[10]: The name of the node. 

//end objid? 


Jf [PRR PER RR RT RR EE ACK EK ER RK A ER AC RC AC ACK CE 2 a a 2 2 2 2 2 a 2 ACK 7 / 


. SUBNET? STATE 


The following code is contained in the subnet? state in the Fibre Channel switch 


FSM. 


Jf [RRR RA RA HK HH HR EE 2 2 2K EE RK EE A A 2K 2 a a EK 2 2K 2K 2K 2 ok a 2K 2 2 3K 2K // 


//Subnet state determines the subnet of the switch 
//This is used for the routing of packets and does not 


//mirror any functionality of Fibre Channel 


subnet_id = op_topo_parent(temp_node_1id); 
subnet = subnet_id; 
printf("My subnet # is: %d \n", subnet); 

pay 


[PEPE TE ER RR OR OR AR DG EE IG 2 2 IE PE 2A AR i AR PR AG SE Ac eo rs tet ot ieee ec ce 


[| Global Variables [| 


[BRE AG A oh ip 2 OG i oS AS Ag 26 AS A AS AS 2 Ae 28 2c oes Eee Ae 2A he 2 AS 2 ig ae ate eta ots Ae 00 Ae hs hs ela cists ok ote nt ata cre ete tate tn 


/| NONE 


[5282 A AR RR A A AC OR A A EEE 2 2 2 E22 EE 2 ER A EE 2 2 2 2 CO 


// State Variables // 


DE i Ee eae a a aa a eS ea Se gee Se St a Saas Sar stegegst eS 


//int subnet: The subnet of the assoc node. 


Jf FE PM oi in A oe An he 2 obs Ar i 2c 2s he Ae 2 AR 2 2s A SRA he hI Ae Pree ia ok che ie eh eae oe oh 


/| Temporary Variables /| 


[PREG i i a A ie oR A AE AG 2 2 Ae AS oe AC RS Rte Aztec chet cae toe tect cto gto etch chee ate cia ge eet he 


//Objid* subnet_id: Temp holder for the’subnet Objid. 


//end subnet? 


jf [RE RE Ae Ae EO ER 6 2 Be 2 ae OE oh 2 oR oe AE 2 2 ae Aa ok oe ee EO 2 ie oo ee ok a Ok ok ee 


oF CON_OUT STATE 


The following code is contained in the con_out state in the Fibre Channel switch 


FSM. 


If [ EPR A HR AH A RE AE AE A EO 2 EE 2 2 2 2 2 oe 2 2 2 ee 2 ee 2 i a 2K / 


// This module finds all the links attached to the switch 
//and builds a look-up table so when a packet is routed 


//to a node, the switch can quickly find the outstream 


for ( j=0; j<= 31; j++) { 
sprintf(xmt_name, "xmt%d", j); // all the xmitters have a std name 
printf("xmt_name = %s \n" ,xmt_name); 
/now we find the objid of the xmitter and save it 
xmt_id[j]=op_id_from_name(s_id, OPC_OBJT YPE_PTTX, xmt_name); 
printf("xmt_id is %d \n",xmt_id[j]); 


//find the objid of the attached link and save it 
link_id[}] = op_topo_assoc( xmt_id[j], OPC_TOPO_ASSOC_OUT, 
OPC_OBJTYPE_LKSIMP, 0); 
printf("link_id is %d \n",link_idfj}); 


// if the link id is valid (ie if the xmitter is connected to a node) 
//then the node_id is the parent of the receiver connected to the link 
if dink_id{j] != OPC_OBJID_INVALID) _ { 
node_id{j] = op_topo_parent( op_topo_assoc(link_idf}j], 
OPC_TOPO_ASSOC_OUT, OPC_OBJMTYPE_RECV, 0)); 
1 end if ‘ 


printf(""The sid, link and connected nodes are %d , %d , %d \n", 
s_id, link_id[{j], node_id[j]); 
} 


ee EE ee Eh ee ee ee ee ee ee eee) 


/| Global Variables // 


URS SE Se aa So SS St ee ce ea Se Ss Se eS er Sted Seatac SE Sect Se ceca gees ce Se Se See ee Sout Fe 2 


// NONE 


fs i rhs he 2 oe oh ole HR Ae oho vhe ho ls is Ao os ols 2c ahs ha rts sha 2 Ae hobs 2h ots he ke As oh ok As 24s oh 2s 2h rhs he 2 As 2 hs ote 1 2h che a ois i a ee 2 ok 


// State Variables // 


1S a rea ae ee i tS ie ee St cca Ia sao Pade sc ea a Aaah Sea at Se at tates Sra deste sean a ean | 


/fint xmt_id[32]: Obyid’s of all the xmitters on the switch 
//int node_id[32]: Objid’s of all the nodes attached to the switch. 


//hnt link_id[32]: Objid’s of all the links attached to the switch. 


| aaa ah Sa et Ea a eal a a a a Sch kati s east atictasins csc Sa Se SO Sec ata 2 


// Temporary Variables // 


[A DH ER RC RC RC ARC RC ACHR ICR RCO RC OR A CO AC RE 2 2 IE 2k 2 oe 2 oo a 2k 2K / 


//char xmt_name[5]: Name of the xmitter on the switch. 


//end con_out 
Jf [PEPER A A He ee 2 2 2 HS AC A 2 a ak 2G 2 2 ok 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2k 2 2 2 2 a 2 OK 
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D. SEUNG eOsT STATE 


The following code is contained in the set_link_cost state in the Fibre Channel 


switch FSM. 


J [FO RAE AE 2 A AE SE OE Oe Ae ee 2 2 AE A oe aS Ae ots Ae As hh a he eA eee 


//set_link_cost 1s responsible for keeping the "cost" 
//associated with traversing a link. This will ensure 


//that over time the traffic is balanced. 


op_intrpt_schedule_self(op_sim_time() + DEL_COST, 0); //schedule the next intrpt 


for (i = 0; 1 < 32; i++) { 
//printf("“link_id %d\n",link_id[1]); 
if (link_id[i] '=OPC_OBJID_INVALID)  { 
util[{i] = (total_bits[i]/(op_sim_time() - START_TIME))//BAND_WIDTH; 
op_ima_obj_attr_set (link_id[ij, “cost”, utl{i)); 
//printfC"util is %f \n" util [i)); 


[EEF EA AR A AEE A AE AE 6 oe oe 2 Ae IR ia Re oto hc oe aie to tee ie 2 he a che oe ote As Ia 2 oh oh Ae oh 


// Global Variables // 
Gf FA ee AAs 2A Ao Ae He He 2 AE he he He he Aa a AS ee Re AS 2 he Ae 2 AA he de a he A= Ae eee le oh he A Ae ta he a i ar Ei 
//double START_TIME: The time the simulation started. 

//const double DEL COST: How often the cost is updated. 

//const int BAND_WIDTH: _ The bandwidth of Fibre Channel. 


[3 PR 2 He A He A Ae A ee A AE AG AC AS A 2 Ae 2 2A 2 2A 2 2 Ee 2k ee 2 2 2 ke 2 2 EE 2 AC Ra A 2 A Ea AE / 


// State Variables UE 


[FEAR RAR A A He Ae He Ae He A eK Ae OK Ae Oe Ae He Ae ee He 2 AE He 2 2 he A he 2K AR 2 oe 28 6 OR 2 AC He 2 OE Ae 26 2 2 OE Ae ae 2 OA Ae A 2 ef 


//float = util[32]: Utilization of the link. 
//int link_id[32]: | Obyjid’s of all the links attached to the switch. 
//int total_bits[32]: Total number of bits to pass through the assoc link. 
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[9H RRR A A HE A A ACK HH HH HH HH HE HA A A ACA AC ACCC EACH CAC 2 CH HCE SK OK OK / 


// Temporary Variables // 


[EEE RK RR HE HE TT EH A A AC CIC EE HE SAIC AE AER ACR ACR ACK 2 A AC A Ao ok 2 oo 


// NONE 


//end set_link_cost 


Dacca ke AG 2h ok Ae rE Ae a ea ie Ia ara ie i oR ee Ba eee er er ie a iT Oe a Pea OE ok CE Oe Cee | / 


10. TYPE STATE 


The following code is contained in the type state in the Fibre Channel switch 


FSM. 


Fp ke Heit ie 2h ole ey he 2h AC cas ie ote eis He es oleh i hae i he a hte oh tate a Rate ee Fetch ta dence eects nec coh ete ences ric ote cto oh ae 


/{Type state determines if the packet is a primative or a frame 


streamIn = op_intrpt_strm(); //which stream did it come from? 


pkptr = op_pk_get (streamIn); 


//if numFields < 3 then its an r_rdy 

//else if numFields > 3 its a frame 

op_pk_format_info_get(pkptr,OPC_PK_PROPERTY_NUMBER_FIELDS, 
&numFields); 


Fhe teeke be be 2 abe 2h be 2 OFS he fe he Ae oie Ae Pe Aa ch ie i on Re oe ote ha ote Ee he ete ele oh ae ett tte tee) 


// Global Vanables // 


[5A BR EE A SEK RRR ACI ER EEE AEE ES SSI SR RRC AC RCRA RCAC OR A EK a 2 2 2 2 2 Oo aK / 


// NONE 


JE a eho Sas Seas costae St hae eats Scot SE St Sc Sash Sask Se Seock Se Se eck Se See oe SS se es a Se Set ea Sc ota ga | 


// State Variables // 


[ERK AR EH EE A RC RC IC AC 2K 2 2K 2 A OK a 2K 2 a a 2 a 2 aK a of 2 oi 28 oi a 2 2 a EK / 


// NONE 
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J FR PRE AG 2 6 Ae oh iG ae ois ls oie AS AG ae ote AS os oe 2 os at cla ag sah ins oho is 2h hs os Ar os obs oh le he oh of 2 he oie 2k oh ke oI ie 2 feb 2 Neo oho a re 


// Temporary Variables // 
[PRR Te ei i Ae eich EAS ACs 2h A I I ia etal te ehh ohh ote ote Eek he che he oo ota net te a oh eth eh 
//Packet* pkptr: The pointer to the active packet. 

//int numFields: |§ The number of fields in the frame. 

//int streamIn: The stream that the packet came in on 

//end type 


ff Fr PEFR A OR Ae Ae Ae A SAC a a Ra ies a AG AC Ae As hee ie Ac ho he he tn ao abe ea oho he Ie ae hs 22h Ae Ae oie hee ht ee 


le R_RDY STATE 


The following code is contained in the r_rdy state in the Fibre Channel switch 


SIV 


J [FREE EAA ABT I A eB As ts Re le SN he aetna sche hy a tha rs tho rhe Ae ahs lee ole He oie fe ob oe ok ee oe oe he ate ohh ta ee 


//\f you got an rrdy increment the b2b credit 


//get the src node from the r_rdy and put it into src_nde 


op_pk_nfd_get (pkptr, "src_nde", &src_nde); // where did it come from? 


inc_buf = get_outStream(node_id, link_id, src_nde); //increment the buffer assoc 
//with the incomming rrdy 
if (buf_to_buf[inc_buf] < max_credit) { 
buf_to_buf[inc_buf] == buf_to_buf[inc_buf]++; 
\ else { 
buf_to_buf[inc_buf] = max_credit; //can never be > than max_credit 


op_pk_destroy(pkptr); 


PPE EE Ae EO EG oF He ete ate ae Ae ie he A Ae ein Ae As Heats he ole che the meee he eee Ae Me Phe ee A ie ee 


// Global Variables // 


[RAE AE As ARO A A He eee OS OR HA 2 Ne oe i Ae 2 Ae 2 2 2 Oh He Ae Oe 2 Ae 2 Oe He Ae ee 2 Oe ee He Ie he ee He ee 2 ee Oe OF oe Oe oe oe 


// NONE 
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[957 2 EE ACH CC AC 2 A A 2 A oR A EE 2 2 A EE 2 2A EE 2 2A 2 oe oe 2 2 2k oe 2 / 


/| State Variables // 


[2898 RI A AR A HC OC A HACC a EC CR A A A EE A EE 2 2 CE EE EEK / 


/int  buf_to_buf[32]: Holds the buffer to buffer credit of each attached node. 
/Ant node_id[32]: Objid’s of all the nodes attached to the switch. 


/Ant —link_id[32]: | Objid’s of all the links attached to the switch. 


/int max_credit: The login buf_to_buf credit. 


cia rarest Re 2h he oe 2b AG he Asati eae Fae 2h hs ake HS 8 oe oh He ie She ie crs eke ok che oe OR iake ie aks 2 fe She ie oh ote coke ee ols ak ots ke 2 eb eee 


/ Temporary Variables // 
rast Ache oh Ae ee 2h he Ae He 1 eR eR a oi oe he oie oR a He A a Ac Ae Ee ae 2 oi OR 2 Oe no oe oi oe oe AE “EE oe oc 
//Packet* pkptr: | The pointer to the active packet. 

pmb src_nde: The node that just sent the packet. 

// This 1s where the r_rdy is returned to. 

//int inc_buf: Inc the buf of the node that sent the r_rdy. 

//end r_rdy 


Uf [PERE Fee Fie i ae Ae IG he 26 OIG 2G hc ie AG 2 2 22 2h OAS I Fe AG hs Oke he he he fe Sie hs of oe ie Ae 2 Ae fe Ae oft ake 2c ho ake oA ois ia ee ele eo ee Oe eek | / 


12. FRAMESTATE 


The following code is contained in the frame state in the Fibre Channel switch 


FSM. 


Fh f Ie Ae Ms hs Aah As oka ats AS AS obs hs AS obs AS AS ats 21S oho AS fe 246 AS hc AC ats eke he AS os 2 AG OAs obese hehe rhe aks ots ola tp ots ohsicts eke cfs ots oh aks hearts cha ciate ety che ta ta tects elo elo rie vhs / / 


//Frame advances the route pointer and helps with debug 


op_rte_pk_advance(pkptr); 


op_rte_pk_src_node (pkptr, &subnet_id_ptr, &node_id_ptr); 
printf("My source is (%d, %d) \n", subnet_id_ptr, node_id_ptr); 


op_rte_pk_dst_node (pkptr, &subnet_id_ptr, &node_id_ptr); 
printf("My destination is (%d, %d) \n", subnet_id_ptr, node_id_ptr); 


eng) 


op_rte_pk_next_node (pkptr, &next_subnet, &next_node); 
printf("I am the switch and my number is %d \n",s_id); 


printf("The next node is: %d \n", next_node); 


[5 RR HR AR RR RR OR RRR CR ECR RR RC KK KK / 


// Global Variables // 
// NONE 
// State Variables // 
// NONE 
// Temporary Vanables // 
/[Packet* kptr: The pointer to the active packet. 
Pp 
//int® subnet_id_ptr: Used to temp hold subnet id's 
//int* node_id_ptr: Used to temp hold node 1d’s 
//int* next_subnet: Used to temp hold the next subnet 
A Vl a next_node: — Used to temp hold the next node id 


//end frame 


Jf [RRR RRR RRR RR RH RR RRR A ROR RR RRR RR RR CR CR RR RRC a ACK 


13. DEC_BUFSTATE 


The following code is contained in the dec_buf state in the Fibre Channel switch 


FSM. 
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J [RRR AA A HH ACA A A AH AC ACE ACH ACR A AC 2A 2A A EEE 2 EEC A 2 2 RO 9K OK OE OE OOK / / 


//Dec_buf state checks to ensure that there is credit with the next node on the route. 


//If there is a busy fabric, statistics are recorded. 


//Ensure the correct transition to the next state 
credit = -1; 


lasso. 1° 


// where is it going to? 


dec_buf = get_outStream(node_id, link_id, next_node); 


if (buf_to_buf[dec_buf] >= 1){ // if you have credit with the receiving node 
buf_to_buf[dec_buf] == buf_to_buf[dec_buf]--; 
//decrement the credit in that buffer 


Sieait—ale //transition to snd_rrdy 
} elSC uy //no credit with the next node indicates a busy fabric 
op_pk_nfd_get(pkptr,"SOF_3",&classType); //what class is the frame, 
//no f_bsy to class3! 
if (classT ype == 0x56) { /Nf it’s aclass 3 destroy the packet 
//and wait for the next frame to arrive. 
op_pk_destroy(pkptr); //destroy the packet 
DED_CL_3 = DED_CL 3 ++; //increment the number of class 3’s 
//destroyed by this switch 
op_stat_wnite (ded_cl_3_gsh, DED_CL_3); //wnite the stat 


classs — i: //return to idle 
/| I have no credit and I’m aclass 1/2 frame 
} elseif (Classy pe == Oxon classic == 0x55) { 
credit = 0; //g0 to f_bsy 


IS 


[7875 FH HH RRA HE AAC EE HEH RK KR KE A CR A EE 2 EC 2 EE 2 EE 2 2 a a 


// Global Vaniables // 


[RE RT A HO RR ACHE EEE RACE OR RR CC 2 CEE 2 RE 2 A a 2 a a a RC ACK 


//Aint DED_CL_3: Stat for number of class 3 frames destroyed by the switch. 


[FBP AE AR OB AE Ee a ee Eee EA hs IS 2h Ae Ae oe nhs 2 Ae 2 A a ee ee 


/ State Variables H 


[PEPE AER EEE AE Ae 2 A eA Ee aie AE 2h oe ee Ae oe Ae 2 ee te ot at oe ee ch ace Ae ee 


//Stathandle ded_cl_3_gsh: SH which holds the value of DED_CL_3 


//int node_id[32]: Objid’s of all the nodes atch to the sw. 

//int link_id[32]: Objid’s of all the links atch to the sw. 

[FRR FE Ee EE AC Ae Hee aise Be he oo ete a ae Aa He AG Hee 256 6 AS Ae aR he Aa ote Aa te Aes Pen Aa ate athe ie ita ha 
// Temporary Variables // 
[FE ARE AR A AE Ae Ae Ae he He Ae RAs ie OR AG oe eA rs 2 AG oe ie AR GAC 2 he hah A Aa ra rat a ee 
//Packet®* pkptr: The pointer to the active packet. 

/Ant* next_node: Used to temp hold the next node id 

//int credit: Used to help the transition to f_bsy or sndrrdy. 

/Ant class3: Used to transition to idle if no credit and a class three Frame. 
//int classType: Holds the class of the Frame. 

//int dec_buf: Index to the buf_to_buf array. Tells which buffer to reduce by one. 


/fend dec_buf 


J [PEER a BAA eR oe he TS Sha he Hs oho ois he hs Ae 2 He Fe Ae 2 AG AS ie AS ris Ae Hs 7k is 2B OF 2 oie ote he he oe oR fe koe ore 2 IR oi ch oe oe ea 


14. F_BSY STATE 


The following code is contained in the f_bsy state in the Fibre Channel switch 


FSM. 


[ [PER RAR AA Ae EEE RE OE A eo 2 BA Oe AE Ae 2 A Ae She Ie 2 2 OK ae 2 2 26 76 Ie 2 A OK RK OK oe ee 


//f_bsy determines if it is an F_BSY to data or to a LCF 


//find out if it’s a FC-O (LCF) or FC-1 (DATA) 
op_pk_nfd_get(pkptr,"R_Bits", &fc_type): 
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//swap the D_ID/S_ID so you can return it 
op_pk_nfd_get(pkptr, "S_ID", &temp_sid); 
op_pk_nfd_get(pkptr, "D_ID", &temp_did); 
op_pk_nfd_set(pkptr, "S_ID", temp_did); 
op_pk_nfd_set(pkptr, "D_ID", temp_sid); 


if (fc_type >= 0x2 && fc_type < OxC) { //Af it’s a data frame 
op_pk_nfd_set(pkptr, “"R_Bits", OxC); //make it an Link Control Frame 
op_pk_nfd_set(pkptr, "INFO", 0x5); //make it an F_BSY to data 
op_pk_bulk_size_set(pkptr,0); //make the data size zero 


} else if Gemmype —— Ox€) { //if it’s a link control frame 
op_pk_nfd_set(pkptr, "INFO",0x6);  * //make it an F_BSY to LCF 
op_pk_nfd_set(pkptr, "RX_ID", 0x00); //give it an exchange ID 
} 

/ Global Variables // 

// NONE 

PFA AA Ae At eek oh As Ae RAs Ae as Ae fe oie Ae he 2 Ac Ae Ahh A A ee ek ee ee FO oe ae A a Ae Ae As He Hh A ee 

| State Variables // 

[EEA AA a AE EE ek Ee EE ee a ee a ET 

// NONE 

[2 9 2 FE eR EE Ee 2 EE 2 2 2 2 2 2 fe 2 2 2 FE 2 2 2 fe 2 2 2 2 2 2 2 2 ae 2 2 2k 2k 2k 2 2 a 2 2 2K / 

/ Temporary Variables // 

//Packet* pkptr: The pointer to the active packet. 

//int fc_type: Data type or Link Control Frame. 

//int temp_sid: Temp holder for the s_id. 

/fint temp_did: Temp holder for the d_id. 

/fend f_bsy 


IffPR PE PRR AR A ER RE A AC 2A A AC OR A 2K A A EE A EE 2 a 2 2 oe a 2 2 2 aC 2 2 iE 2 2 iE 2 2 // 
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>: ADD_ROUTE STATE 


The following code is contained in the add_route state in the Fibre Channel 


switch FSM. 


f [PEAR RE ea Me Ae aR Heo Ao Oe OR oh a ee A a oe oi oe Ae ate Be ate oe oh oho ote Ac oh oe ote eo oh i ok oe eo os te oe 


//add_route adds a return route to the F_BSY generated by the switch 


op_pk_nfd_get(pkptr, "D_ID", &d_id); //get the destination 


dst_subnet = op_topo_parent(d_id); //get the subnet of that destination 


rset_ptr = op_rte_routeset_create (TOPO_PTR, subnet, s_id, 
dst_subnet, d_id, MAX_HOPS); 
tmp_rte = op_rte_routeset_mincost (rset_ptr); 
rte_arry = op_rte_route_copy (tmp_rte); //copy to a temporary route 
op_rte_pk_route_insert(pkptr, tmp_rte); //add the route to the packet 


op_rte_pk_next_node (pkptr, &next_subnet, &next_node); //find out the next node 


[FPR RR ae a ole oie ai ie ake Ms Fe cte che hc cle hy Hehe 08s lp os ois oie as A ate ag aie Aa oe et oA ie a te 2 Oe ee 


// Global Variables // 
[PR PE PR Fi A AE a Ae Ae He As is a OR Oe Ae is obs ee AG Oe oi Ae os hs he oA os oe 2k oe ae AG AR ob sis As AS 2 he As ofS 0 ie A hs Of oF ots Oe Ae ole fe ots ois ohn oe Os oh eta 
//extern Topology* TOPO_P: The one and only Topology pointer. 

//extern const int MAX_HOPS: Max number of hops a packet can take. 


[PRR SAG A BAG Ba OF ahs Sag wie As ols oe Ns ie okey As ols he he ie “Ee ots he AG oie 9 Ae cis ake Ae Ae ke hohe ea le ate Ae os ois i Et ae 


/| State Variables vy 


[FPR AE EB EE eR ee OE a Oe OR 8 2 Ee Oe SE i 2 ae eG EO a 2Ae eo a oe oe a ee ie oa a le te oe oo ee ine eee 


//Route_Set* rset_ptr: The route set pointer for returning packets. 


//Route* rte_arry: The route for the fbsy from the switch to its destination. 
//int S_id: The source id of this node. 
//int subnet: The subnet of the assoc node. 


[5 EAR A Ae ee 2 ie 2 EAC 2 ke 22 2 2 CC CC 2 Ee 2 2 2 2 2 fe E28 ee 2k 2 2 fe iE 2g 2 2 eke 2 2 oe 2c 2k 2 a a 2K / 
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// Temporary Vaniables // 


[RRR RR A KK RK HH HK HI AK A HA A AK A A AC A CAC EE ACE 2 A CC EE 2 / 


//Packet* pkptr: The pointer to the active packet. 

//Route* tmp_rte: The route which is inserted on the packet. 
/hnt* next_node: Used to temp hold the next node id. 
/fint next_subnet: The subnet of the next node. 

//int Geid: The destination. 

/Ant dst_subnet: The subnet of the destination. 


//end add_route 


Ue ek ee ee A EE EEE EE EE Ee | / 


16. SNDRRDY STATE 


The following code is contained in the sndrrdy state in the Fibre Channel switch 


FSM. 


fap te ote Ae Ae 2s Ak oe AG ahs 26 OFS 2 AC RG AS Se AG AS AS SG 2h AG of A ie 26 AG oR AG PEG AG A I 2A AC ORS he 2 ote AS AR Ae oA Rh eee ef 


// snd rrdy builds an rrdy and sends it back to the previous node that sent the frame. 


rdyptr = op_pk_create_fmt ("R_RDY"); 


//get the src node from the frame and put it into src_nde 
op_pk_nfd_get (pkptr, "src id", &src_nde); 

printf("the previous node is %d\n",src_nde); 
printt("MY NODE IS %d \n",s_id); 


//put your s_id in the "src nde" field 


op_pk_nfd_set (rdyptr, “src_nde”, s_id); 


// Find the packet stream connected to src_nde 

streamOut = get_outStream(node_id, link_id, src_nde); 

printf("rrdy streamOUT = %d\n", streamOut); 

printf("1'm going to = %d, nde_ary = %d\n", src_nde, node_id{streamOut]); 
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//the following data will be used to calculate utiliztion 


total_bits[streamOut] = total_bits[streamOut] + op_pk_total_size_get (rdyptr); 


i set some delay based on the time it took to get to this point 


// i.e. had to process all the stuff up to now. which took time. 
// if one wanted to have some PDF based delay this would be 


// where that delay would be placed. 
// for example op_pk_send_delayed(rdyptr,streamOut, 
// op_dist_exponential(r_rdy_delay)); 


if (r_rdy_delay '=0) { /Nif there is a non-zero delay 
//send it out the transmitter with the proscribed delay 
op_pk_send_delayed (rdyptr, streamOut, r_rdy_delay); 
} else { 
//send it with no delay 
op_pk_send (rdyptr, streamOut); 
} 


J RR OR He Ae TS 2% AG AG ie 2 8 hoe AG AG ote ie oBn ots 24S AS 26 FR oh he oe 28 AS op he obs he te ke 28 ho ofa Ae ob oh ode Aso oho srs cee te oe ote he 2 Phra as cian rae 


// Global Variables // 


[PEER EE Ee Ee EE oe OR 2 a a Oe Ro ee ee oe ae ek eo Re ec he Re 


// NONE 


J FE 2 2 2 a 2 2 oe ee 2 ose he 2 2 2 6 24 2 2 2s 2 a 2k Ae 2c oe 6 6 fe 2k 2 i 2c 2 2 26 ie 246 2 fe 26 2 246 26 2 2K he 2c 2 he 24e 2 246 2 2 2 OK 2c 2 2k 2c 2k 2c / 


// State Variables // 


[FRR RA He He 2 2 Ae A ie 2 OF 2 AG 2 A He OR 2 Oe AG He 2 Oe 2 2 OF 2 Ae Ae oF he 2 ae Ae 2B 28 fe 2 2 A ee 2 Ae 6 OR 2 Ae 2 2 2s 2 OF 2 OF Ae OE EH 


//double r_rdy_delay: Delay required to process, build and send an R_LRDY. 


//double total_bits{32]: Total number of bits to pass through the assoc link. 
//int node_id{[32]: Objid’s of all the nodes attached to the switch. 


//int link_id{32]: | Obyjid’s of all the links attached to the switch. 


[FEAR Ee a He HK Ae EO A OE OK ee ae AG AC AR he 26 2 ie Ae AE 2 2 Ae a 2 2 OG AS Oe oe 2 2 ie 28 2 2 2 AG AG 2 2 6 AG 2 2 2 EO Ae OE OE EOE 


// Temporary Variables // 


[FR 2 2 ie 2G 2 ee 2 ee oe 2c 2 2c 2 ee 2 26 2 2 2 2c os 2 fe 2 2 2 2c 2 fe 2c 2 9 2 i 2 2 2c 2 2 Ke 2c as 2c 2 2c ake 2c 2 2c 2 24 2k 2c 2 2c ae 


//Packet* pkptr: The pointer to the active packet. 
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//Packet* rdyptr: The pointer used to create the R_LRDY primative. 


/Ant®* SiminGle. The node that just sent the packet. 

/| This is where the r_rdy is returned to. 

//int streamOut: Index of the stream where the R_RDY is going. 
//end sndrrdy 


| [RHR RR I TH EE I A AR RR RR RCE CO EC AR OK AR OR OE RA OK OK OK OK 9K OK ICI AIEOK / 


17. XMT 


The following code is contained in the xmt state in the Fibre Channel switch 


FSM. 


//xmt is used to send the frame to its next destination 
op_pk_nfd_set (pkptr, "src id", s_id); //put your sid on the frame 


//find the stream that has the next node attached to it 
streamOut = get_outStream(node_id, link_id, next_node); 
printf("The output stream index is %d\n", streamOut); 


printf("next_node = %d, nde_ary = %d\n", next_node, node_id[streamOut)]); 


//the following data will be used to calculate utiliztion 


total_bits[streamOut] = total_bits[streamOut] + op_pk_total_size_get (pkptr); 


if (proc_delay '=0) { /Af the delay is non-zero 
//send it out the transmitter with the proscribed delay 
//this would be a good place to input a PDF for proc_delay 


//for now we will use a const proc_delay. 
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op_pk_send_delayed (pkptr, streamOut, proc_delay); 
eised //send it with no delay 


op_pk_send (pkptr, streamOut); 


Yi a a a ae ec Sc a a ae Se Se ee Te ae ae oe aa cas ce ec Sees 


/| Global Variables // 


Je Ae ae ee ASS IC IS Re i a A ia oe le ahaa a AS PIS I 2 2 Ie re AS 2A SA he he ete hehe te ote ate te oka oe ote As oho aie iS oe As ote 


// NONE 


i a a a a a a a a ce a ee aa a ae cas ae ee ak his st SSC A 


// State Variables // 


PRE Ra A ee PR ait Ee hohe rhe cfs Ae oF AR oh ie AS aie eo oie obs Peete Pe che oe ie ete ee che ate he he ee tet ee ce oie eke Pio eieck o 
//double proc_delay: Delay required to process, build and send a frame. 


//int node_id[32]: Objid’s of all the nodes attached to the switch. 


//int link_id[32]: Obyjid’s of all the links attached to the switch. 


//Aint total_bits[32]: Total number of bits to pass through the assoc link. 
[FRAG A A oe AE es oi es Sheri chs ale te toes he Ae i ts AS obs he of oho As As 2s ole is ctasahe chp hob oho hv cha oh ae oh che os th a rhe oh he oe hs hae i 


// Temporary Variables // 
[RFR AR Ay ots AS hn Hs 2 AG oR she ole ie ois obs AG AG 2s As HAS oh hg he oS in a she he lp ls oh hs chs AS ohn rhe op iy cfs oho in he hs rhs ce iach chs ohn oa nh 
//Packet* pkptr: The pointer to the active packet. 

int next_node: — Used to temp hold the next node id 

//int StreamOut: Index of the stream where the R_RDY 1s going. 

//end xmt 


J [PR PO PRA A OS Ee Ie hs a Ae sin 218 AS PAS 28 AS 286 Ae hs PASE AS ots hs els rhe che rhn tas As ole Pecans. sobs hs es hy ris ots te te he OKC 2 nha ehh vis th sche ote eee ete 2k 2s ead 
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l. 


APPENDIX D. MATLAB CODE USED DURING SIMULATION 


LEAST SQUARES FIT OF SETTLING TIME DATA 


function coef = settletime (DEL_COST, Settle_time) 
Omer <= 1 %order of the polynomial 
% Now use LSQ fit to get slope and y intercept 


eq _of_line = polyfit(DEL_COST, Settle_time, order); 


fe ( Olseee 2 rs used to show curve fit for t = 0:2 


eq of _line(1)*t + eq_of_lime(2); % y = mx@eb 


iy 


tTheory shows what the polyfit would get w/ 
%other values of DEL COST 


theory = eq_of_line(1)*DEL_COST + eq_of_line(2); 
% Table to show the data 
table =[DEL_COST Settle_time theory 

(Settle time - theory) ] 


SGiff = absolute difference btwn theory and simulation 
ditt i= abs (tablet. 44) 


%Saverage difference btwn theory and simulation 
mean_diff = mean(diff) 


coef = eq_of_line; 

m=num2str(coef(1)); @Slope of the line 
b=num2str(coef(2)); SY intercept of the line 
figure (1) 


plotted using log-log for clarity 
logleg (ts y, DEEZEOST, Seerle - Eime, “rol 


xlabel('DEL\_COST (sec) = x') 
ylabel('Settling Time (sec) = y') 

legend ([{y =—'4ine x bi) Samu ation pata, 2) 
Sagal 
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ue CALCULATION OF TIME TO R_RDY 


function time = time2rrdy (frameSize, dist, refIndex, 
fara Dy } 
6 time = time2rrdy (frameSize, dist, refIndex, rrdyDly) 


fe)\o4 


oP 


frameSize is the size in Bits of the Data Payload 

dist is the one way distance from the node to the switch 

refIndex is the refraction index of the optical fiber 

rrdyDly is the time required for the switch to respond to 
the frame with an R_RDY. 


oP oP oP cP oP 


oo 


This function calculates the time required for a node 
to receive an R_RDY reply from a switch after sending a 
frame. 


oe 


oP? 


first define the constants 
tEtls = 17 2062500000 - Stime to transmit one bit 


ce) 


propSpeed = 3.0e8./refIndex % speed of light in the medium 
ELaylime. = £elp-~40- S6time to transmit the R_RDY 
framexXmtTime = frameSize*ttlb; %@xmission delay of the frame 


propTime = dist./propSpeed; %Prop delay through the fiber 


S$Total time calculation 
time = rrdyTime + 2*propTime + frameXmtTime +rrdyDly; 
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5). CALCULATION OF REQUIRED CREDIT 


function credit = calcCredit(secPerFrame, returnDelay) 


credit = calcCredit(interarrival_arg, returnDelay) 


oP oo oP 


For this function you should enter the interarrrival 
argument for the secPerFrame and either the time required 
for the node of interest to receive a R_RDY from the 
Switch for a Buffer-to-Buffer calculation. 

Or, for an End-to-End credit calculation use the end to 
end delay through the system of interest. 


0 dD od oP 


oe 


credit = ceil(returnDelay./secPerFrame); 
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INTERARRIVAL ATTRIBUTE CALCULATION 


function iarr = liarrCalc(throughput, dataBitsPerFrame) 
$larr = larrCalc(throughput, dataBitsPerFrame) 


% throughput is the desired output in BPS of the 
$node of interest 

% dataBitsPerFrame is the number of payload bits 
Sthat are expected per frame. 


This function calculates the interarrival argument 
for the ideal packet generator in the node model. 
$This function allows the designer to take a data 
Sthroughput requirement and translate it into an 
Zattribute in the simulation node model. 


$This does the 8b/10b encoding plus header 
bitsPerFrame = dataBitsPerFrame*10/8 + 600; 


This calculates the actual Fibre Channel 

Sthrougput including the overhead of the 

Sheader and that caused by 8b/10b encoding. 

reqtput = throughput. *bitsPerFrame./dataBitsPerFrame; 


disp(’Required Fibre Channel Througput 1s: ’) 
disp (reqtput) 


disp(’This represents a utilization of: ’) 
disp (reqtput/1062500000) 


SFrame interarrival rate 1S one over the regired 


SeEnLOuUgipU & . 
larr = (1./reqtput) .*bitsPerFrame; 
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OPTIMAL CREDIT CALCULATION FUNCTION 


function credit = calcCredit(secPerFrame, returnDelay) 
Scredit = calcCredit(interarrival_arg, returnDelay) 


For this function you should enter the interarrrival 
Sargument for the secPerFrame and either the time 
required for the node of interest to receive a R_RDY 
from the switch for a Buffer-to-Buffer calculation. 
Or, for an End-to-End credit calculation use the end 
Sto end delay through the system of interest. 


credit = ceil(returnDelay./secPerFrame) ; 
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